How-To Tutorials - Web Development

  Joomla! 1.6 First Look A concise guide to everything that's new in Joomla! 1.6. Anyone who's used to working with the previous versions of Joomla! knows the old section—category—article drill. Articles had to be part of a category and categories had to be part of a section. There were no workarounds for this rigid three-level content organization scheme. Sometimes, this required Joomla! users to adapt their content to the system's limitations (or extend Joomla!'s functionality by using more powerful content management extensions, so-called Content Construction Kits or CCKs). In Joomla! 1.6, the rigid old system has finally been replaced. Sections have gone; there are now only categories, and any category can hold as many levels of subcategories as you need. In the backend, instead of both a Section Manager and a Category Manager, you'll now find only a Category Manager. You can forget the concept of sections altogether; in Joomla!! 1.6 there's no need for them anymore, as they're no longer needed as 'containers' to hold categories. Improvement #1: categories can now be one level deep Sometimes, you'll want to organize articles in just one category. Let's say you want to add a few articles about your organization: who you are, where to reach you, and so on. You don't need any subcategories. You'd need a structure like this: In Joomla! 1.5, this simple setup of a "sectionless" category holding articles wasn't possible. You'd have to organize content in sections and categories—which implied that any group of articles would be stored two levels deep, even if you didn't need this. The only alternative was not to organize content, using uncategorized articles. In Joomla! 1.6, you can put content in just one category if you want to. Just go to Content | Category Manager | Add New Category to create a new category. In the Parent drop down box, select No parent: As this category has "No parent", it becomes a top-level or "parent" category. It's as simple as that; now you can do something that wasn't possible in Joomla! 1.5, by assigning articles directly to this category. Improvement #2: creating multiple category levels Joomla!'s old section—category—article approach didn't allow you to create categories within categories ( "nested categories"). However, on content-rich sites, you might need more than two levels of content organization and use a few subcategories. Here's an example from a site featuring product reviews. It uses several levels to organize the main category of "reviews" in subcategories of product types, brands, and models: A great advantage of being able to create such a structure is that it allows for very specific searches (that is, within categories) and multiple ways of navigation. Another example is if you are creating a catalog that you want to be searchable with multiple filters such as manufacturer, price, general item type, or a specific product name. Creating a set of 'nested' categories Let's find out how you can quickly set up a few nested categories like the ones shown in the illustration above: Go to Content | Category Manager | Add New Category. In the Title field, enter Reviews. In the Parent field, make sure that the default option No parent is selected. The screen should look like this: Click on Save & New. A message appears to confirm your action: Category successfully saved. At the same time, all the fields in the Add New Category are emptied. To create a subcategory, enter the subcategory name Cameras in the Title field. In the Parent drop-down box, select Reviews: Click on Save & New to store the subcategory. Repeat the previous three steps to create more subcategories. For each new category, first enter a title, then select the appropriate parent category and save it by clicking on Save & New. When you're done with creating subcategories, click on Save & Close to view the results in the Category Manager. In the example below, the Cameras category is parent to a subcategory Compact Cameras. The Compact Cameras category is parent to a subcategory called Canon. If you've followed the above example, you'll find the following set of categories in the Category Manager. They are displayed as shown below: The Reviews name isn't indented, as it is a top-level category. Cameras, Compact Cameras, and Canon are displayed indented as they are subcategories. When you create articles, you can now assign them to the new categories. The same category hierarchy as you've just seen in the Category Manager is displayed in the Category drop-down box: Using nested categories in the sample data You've just set up a few categories and subcategories yourself. On a complex site, you can have a far more complex structure. Don't worry, I won't ask you to create dozens of nested categories right now—but it's a good idea to learn from the example set by the Joomla! Developers. Let's have a look at the categories and articles that come with Joomla! when it is installed with sample data. The way things are organized there will give you some idea of how you can deploy nested categories and get the most out of the new system. Exploring the sample data On the frontend, click on the Sample sites link in the This Site menu. On the Sample Sites page, a new menu appears. This menu gives access to both sample sites—Australian Parks and Fruit Shop: Have a look around at both example sites. They appear to be separate websites, but they're not. Here the Joomla! developers have cunningly deployed the possibilities of the new category system and have organized all content for the three sites (the main site and two example sites) within one big website. To find out how this is done, let's have a look at the categories in the backend: Go to Content | Category Manager to see how the sample content is organized. The screenshot below shows an overview: As you can see in the screenshot above, there's one top-level category, Sample Data-Articles. All other articles are contained in the subcategories of this main level category. Apart from the top level category, there are three main categories: The Joomla! category. It has three sublevels. The Park Site category. It has two sublevels. The Fruit Shop category. It has one sublevel. Finally, there's a group of articles that's not in any category; it's a bunch of leftovers all marked as Uncategorized. How can different categories look like different sites? As you click through the example sites, not only the content changes; the menu links to each main category (such as the Parks and Fruit Shop category) have specific templates assigned to them. This way, on the frontend, the look-and-feel of the different main article categories are totally different, whereas in the backend, they're just part of one big site. Applying templates to categories can give visitors the impression of exploring a separate set of websites. Although there's no limit to the number of levels in the category hierarchy, even in this rather complex set of sample site articles, categories don't go further than four levels deep. It is possible to make more subcategories, but keep in mind that this means that your content will be stored 'deeper' in the hierarchy, possibly making it more difficult for visitors (and search engines) to find it. One benefit of placing interrelated content under its own main level category is that you can easily unpublish, delete, or archive any content dealing with a specific subject by unpublishing, deleting, or archiving this main level category. That's why the Joomla! developers have chosen to use one top-level category for all sample data. By unpublishing the top level category (Sample Data-Articles), you can unpublish all of the example content in one go. New category settings: notes and metadata When entering or editing a new category, the New Category or Edit Category screen now offer you an area to type notes about the purpose of the category or related items, as well as a place to add keywords and a description (metadata). The Note field (found in the Basic Options section) can be useful to share some extra information about the category with other backend users. For example, you can enter a short explanation about this category ('subcategory of ...'): Adding category metadata In Joomla! 1.5, there was no way to separately enter metadata for category pages. Now, you can enter specific Meta Description and Meta Keywords in the Metadata Options section when creating or editing a category. Another new item in the Basic Options of a category is the Alternative Layout select box. Alternative layouts are an advanced new feature that enable you to select a customized layout for the current category, provided the selected template (or a third-party component) provides these extra layout options. A template can contain so-called template override files, allowing for customized layouts that replace Joomla!'s default views. Using the Alternative Layout select box, you can now select the template override you want to activate for this particular item. To find out more about this feature, have a look at the "Introduction to Alternative Layouts in Version 1.6" document on the Joomlacode site. You'll find it at http://downloads.joomlacode.org/trackeritem/5/8/6/58619/introtoaltlayoutsinversion1-6v2.pdf. Fresh ways to display category contents on the frontend Joomla! 1.6 provides several additional methods to display category contents. They replace the four classic layouts of Category List, Category Blog, Section List, and Section Blog. When creating a new menu link pointing to a category, you are now presented with a slightly different set of Menu Item Types: These are the category views are available: List All Categories is a new view, described below Category Blog was previously called Category Blog Layout Category List was previously called Category List Layout The Blog and List views are basically the same as they've always been. However, these display types now offer new settings that provide more control over the look and feel of the resulting pages. Along with the new List All Categories menu item type, there are also a few new module types that provide you with new ways to display links to categories and their article contents. Let's have a closer look at the new category views. New category view # 1: List All Categories The new category system rationalizes the organization of content, even in large or complex websites. One advantage of this is that you can more easily give visitors (and search engines!) access to all that well-structured content, just by adding one menu link to a main level category. This will allow visitors to easily drill down the different layers (the category levels) of the site structure. To achieve this, the new List All Categories menu link type allows you to display categories as well as their subcategory contents. You can see an example of this menu organization if you select the Site Map link on the This Site menu in the frontend of the sample Joomla! 1.6 content. As we've previously seen, the sample data that comes with Joomla! 1.6 is organized in a structured way. The Site Map link uses the List All Categories menu item type to show all levels in the category hierarchy.  

Installing and configuring BP Courseware As BP Courseware is a plugin that runs atop BuddyPress, we must have already installed, activated, and configured BuddyPress. To install BP Courseware, log in to the WordPress dashboard, hover over Plugins in the left sidebar, and choose Add New. Search for BuddyPress ScholarPress Courseware and install the plugin. BuddyPress Courseware requires the Private Messaging BuddyPress component to be enabled. If you have disabled Private Messaging, you will be prompted to enable it before activating BuddyPress Courseware. To enable Private Messaging, log in to the WordPress dashboard and visit the BuddyPress Components screen. BuddyPress Courseware settings While BuddyPress Courseware will work immediately, there are a number of settings that can be adjusted to ensure that the plugin meets our needs. To access the BuddyPress Courseware settings, log in to the WordPress dashboard, hover over BuddyPress in the left sidebar, and select Courseware. Global settings BP Courseware integrates with BuddyPress groups for course management. By default, we must enable courseware individually for each group. Checking the Enable Courseware globally checkbox will turn courseware on for all new groups. This is useful if groups are used exclusively for course management. If you intend to use BuddyPress groups for other purposes, such as student project collaboration, the Enable Courseware globally option should remain unchecked. In this scenario each group will require enabling courseware manually. To do so, follow the instructions given in the Enabling BP Courseware section later in the article. Collaboration settings Within BP Courseware we are able to define users as either teachers or students. By enabling the Collaboration settings option, any site user with a teacher role has the ability to edit and add courses, assignments, and schedules. Make assignment responses private When students submit an assignment their response is public to all site users. By enabling the Make assignment responses private feature, student responses will be visible only to the teachers and the student who has completed the assignment. Gradebook default grade format The default BuddyPress Courseware Gradebook format is numeric. Within the Gradebook default grade format settings we are able to choose between numeric, letter, or percentage grading for assignments. Webservices API integration BP Courseware has the ability to integrate with WorldCat and ISBNdb web services to aid in locating books and articles. To integrate these services with BuddyPress Courseware, follow the links from the BudddyPress Courseware settings screen to sign up for a free API key. Customization Cascading stylesheets (CSS) are the files that control the look and formatting of a web page. BuddyPress Courseware allows administrators with advanced web skills to create a custom stylesheet for fine grain control over the look of Courseware. Renaming the groups page BP Courseware utilizes the BuddyPress group feature for course content. While the term Groups makes sense in the context of a standard BuddyPress installation, it can be confusing when using BuddyPress Courseware as a learning management system. To prevent this confusion, I find it helpful to rename the Groups page to Courses.   To rename Groups: Log in to the WordPress dashboard. Create a new WordPress page titled Courses by hovering over Pages in the left sidebar, and choosing Add New. Title the page Courses, leaving the page content blank, and click on the blue Publish button Adjust the BuddyPress page settings by hovering over BuddyPress in the left sidebar and selecting the Pages tab. In the menu next to User Groups, select the Courses option and click on the Save button. Delete the Groups page by clicking on Pages in the left sidebar. From the All Pages screen, hover over the Groups page and click on the red Trash link. Setting Courses as the site home page Using the BuddyPress Courseware plugin, we may wish to enable our Courses page as the site's front page. Doing so will allow students to quickly access course information and prevent confusion regarding how to find the courseware dashboard.   To enable the Courses page as our site's home page: Log in to the WordPress dashboard. If you have not already done so, create an empty page titled Blog Hover over Settings in the left sidebar and select the Reading tab. From the Reading Settings screen, select Front page displays | A static page option. Select Courses from the Front page menu choices and Blog as the Posts page Creating a course When setting up a course we must first create a BuddyPress group. To create course or group: From our public facing site, visit the Courses page (or Groups if not renamed). Click on the Create a Group button. From the Details screen, provide a Group Name such as the course name and section number and enter a Group Description such as the course catalog information. On the Privacy Options page, select This is a public group, allowing any site member to join. Complete the installation by optionally adding an avatar image and inviting members. Enabling BP Courseware Once the course group has been created, we may enable BP Courseware. This step may be skipped, if we selected Enable Courseware globally from the BuddyPress Courseware settings screen. To enable BP Courseware: Visit the page of the newly created group. Click on the Admin tab Click on the Courseware link from the row of links below the Admin tab. Below Courseware Status , select Enable to enable BP Courseware for the group. Optionally, if you wish to keep student assignment responses private, select Enable below Private Responses. Click on the Save button. Courseware dashboard Within BP Courseware, the courseware dashboard acts as the course home screen for both instructors and students. From the courseware dashboard, instructors are able to add and manage course content. Students use the dashboard to access course materials and submit assignments. To access the courseware dashboard, visit the Courses page (or Groups if unchanged) and click on the Courseware link located below the course/group description. The teacher dashboard appears as shown in the following screenshot: The student dashboard appears as shown in the following screenshot: Adding course content Adding course content to BP Courseware allows educators to easily organize and share course information with students. The BP Courseware plugin provides a structure for managing course lectures, assignments, quizzes, grades, resources, and schedules. Lectures Adding lecture information allows instructors to share course notes, resources, and slides with students in a structured format. While the term lecture implies the lecture format of university courses, I find it useful to think of lectures in terms of teaching units. The lecture pages can serve as a resource for a chunk of course content. To add a new lecture, click on the Add a new lecture button from the course dashboard. We may then edit the content of the lecture much in the same way as a WordPress post adding text, images, links, and embedded media. Lectures will appear in the Latest Lectures section of course dashboard, with the most recently posted lecture appearing at the top. This allows students to quickly access the most recently posted course lecture. Assignments BP Courseware provides a means for us to post assignments and collect student responses. The assignments can take multiple formats, allowing students to respond to questions, upload a file, or embed media. Posting assignments as a teacher As a teacher, we have the ability to post assignments. To post an assignment, visit the course dashboard and click on the Create an assignment button. This will take us to the New Assignment screen. From the New Assignment screen, we may enter the details of the assignment. From the New Assignment screen, assign a title to the assignment and enter the assignment description and necessary information. This acts much in the same way as WordPress posts, allowing us to enter text, images, media, and links. In the right column, we may choose the corresponding lecture topic and select the assignment due date. Once the assignment information is complete, click on the Publish assignment button to make it visible to students. Submitting assignments as a student Students can access assignments by clicking on the All Assignments button or viewing the Latest Assignments from the course dashboard. By clicking on the Assignment Title, students can see the detailed assignment information written by the instructor. Clicking on the Add a response button, will give the students an opportunity to submit the assignment. For standard assignments, students will be presented with a text input area similar to the WordPress post editor. Within this text area, students may type a written response, add links, embed media, and attach files. Viewing student responses Instructors have access to all student assignment responses, while students are only able to view their own. To view student assignment responses, visit the individual assignment page and click on the Responses button, located in the right sidebar. This will present us with a list of student responses. Clicking on the individual assignment link, will take us to a screen containing the student's assignment response. Gradebook BP Courseware provides us with a Gradebook feature for assessing student assignments. To access the Gradebook, visit an individual assignment page and click on the Gradebook button, located in the right sidebar. This will take us to the Gradebook screen for the assignment. For each student, we are given the option to enter a grade value, a private comment, and a public comment. Once the grades and comments have been entered, we may click on the Save grades button. After an assignment has been graded, students will receive a message containing the grade. Students will be alerted to this message by a notification, but may access their grades directly by visiting the individual assignment page. The grade is posted in the Assignment Meta sidebar of the assignment page. Bibliography The Bibliography is designed so that educators can easily maintain a list of course materials and resources. To add entries to the bibliography, visit the Courseware dashboard and click on the Manage bibliography button. From the Bibliography page we may add entries manually or import them by pasting information from BibTeX. The bibliography is ideal for courses utilizing a wide range of materials. BibTeX is a tool for formatting references. It is typically used in conjunction with the LaTeX typesetting system. More information can be found at http://www.bibtex.org. Schedule and calendar The BuddyPress Schedule page functions as a course calendar, automatically containing assignment due dates as well as manually managed schedule items. To view the course calendar, visit the Courseware Dashboard and click on the Schedules link. To schedule more items, click on the Add a schedule button from the Courseware dashboard. Complete the New Schedule form and click on the Publish Schedule button once it is complete. New schedule items will be added to the calendar as well as featured in a list format on the Schedules page.

In this two-part article series, we're building an internet-connected garden bot using JavaScript. In part one, we set up a Particle Core board, created a Johnny-Five project, and ran a Node.js program to read raw values from a soil moisture sensor. Adding a light sensor Let's connect another sensor. We'll extend our circuit to add a photo resistor to measure the ambient light levels around our plants. Connect one lead of the photo resistor to ground, and the other to analog pin 4, with a 1K pull-down resistor from A4 to the 3.3V pin. The value of the pull-down resistor determines the raw readings from the sensor. We're using a 1K resistor so that the sensor values don't saturate under tropical sun conditions. For plants kept inside a dark room, or in a less sunny climate, a 10K resistor might be a better choice. Read more about how pull-down resistors work with photo resistors at AdaFruit. Now, in our board's ready callback function, we add another sensor instance, this time on pin A4: var lightSensor = new five.Sensor({ pin: "A4", freq: 1000 }); lightSensor.on("data", function() { console.log("Light reading " + this.value); }); For this sensor we are logging the sensor value every second, not just when it changes. We can control how often sensor events are emitted by specifying the number of milliseconds in the freq option when creating the sensor. We can use the threshold config option can be used to control when the change callback occurs. Calibrating the soil sensor The soil sensor uses the electrical resistance between two probes to provide a measure of the moisture content of the soil. We're using a commercial sensor, but you could make your own simply using two pieces of wire spaced about an inch apart (using galvinized wire to avoid rust). Water is a good conductor of electricity, so a low reading means that the soil is moist, while a high amount of resistance indicates that the soil is dry. Because these aren't very sophisticated sensors, the readings will vary from sensor to sensor. In order to do anything meaningful with the readings within our application, we'll need to calibrate our sensor. Calibrate by making a note of the sensor values for very dry soil, wet soil, and in between to get a sense of what the optimal range of values should be. For an imprecise sensor like this, it also helps to map the raw readings onto ranges that can be used to display different messages (e.g. very dry, dry, damp, wet) or trigger different actions. The scale method on the Sensor class can come in handy for this. For example, we could convert the raw readings from 0 - 1023 to a 0 - 5 scale: soilSensor.scale(0, 5).on("change", function() { console.log(this.value); }); However, the raw readings for this sensor range between about 50 (wet) to 500 (fairly dry soil). If we're only interested in when the soil is dry, i.e. when readings are above 300, we could use a conditional statement within our callback function, or use the within method so that the function is only triggered when the values are inside a range of values we care about. soilSensor.within([ 300, 500 ], function() { console.log("Water me!"); }); Our raw soil sensor values will vary depending on the temperature of the soil, so this type of sensor is best for indoor plants that aren't exposed to weather extremes. If you are installing a soil moisture sensor outdoors, consider adding a temperature sensor and then calibrate for values at different temperature ranges. Connecting more sensors We have seven analog and seven digital IO pins on the Particle Core, so we could attach more sensors, perhaps more of the same type to monitor additional planters, or different types of sensors to monitor additional conditions. There are many kinds of environmental sensors available through online marketplaces like AliExpress and ebay. These include sensors for temperature, humidity, dust, gas, water depth, particulate detection etc. Some of these sensors are straightforward analog or digital devices that can be used directly with the Johnny-Five Sensor class, as we have with our soil and light sensors. The Johnny-Five API also includes subclasses like Temperature, with controllers for some widely used sensor components. However, some sensors use protocols like SPI, I2C or OneWire, which are not as well supported by Johnny-Five across all platforms. This is always improving, for example, I2C was added to the Particle-IO plugin in October 2015. Keep an eye on I2C component backpacks which are providing support for additional sensors via secondary microcontrollers. Automation If you are gardening at scale, or going away on extended vacation, you might want more than just monitoring. You might want to automate some basic garden maintenance tasks, like turning on grow lights on overcast days, or controlling a pump to water the plants when the soil moisture level gets low. This can be acheived with relays. For example, we can connect a relay with a daylight bulb to a digital pin, and use it to turn lights on in response to the light readings, but only between certain hours: var five = require("johnny-five"); var Particle = require("particle-io"); var moment = require("moment"); var board = new five.Board({ io: new Particle({ token: process.env.PARTICLE_TOKEN, deviceId: process.env.PARTICLE_DEVICE_ID }) }); board.on("ready", function() { var lightSensor = new five.Sensor("A4"); var lampRelay = new five.Relay(2); lightSensor.scale(0, 5).on("change", function() { console.log("light reading is " + this.value) var now = moment(); var nightCurfew = now.endOf('day').subtract(4,'h'); var morningCurfew = now.startOf('day').add(6,'h'); if (this.value > 4) { if (!lampRelay.isOn && now.isAfter(morningCurfew) && now.isBefore(nightCurfew)) { lampRelay.on(); } } else { lampRelay.off(); } }); }); And beyond... One of the great things about using Node.js with hardware is that we can extend our apps with modules from npm. We could publish an Atom feed of sensor readings over time, push the data to a web UI using socket-io, build an alert system or create a data visualization layer, or we might build an API to control lights or pumps attached via relays to our board. It's never been easier to program your own internet-connected robot helpers and smart devices using JavaScript. Build more exciting robotics projects with servos and motors – click here to find out how. About the author Anna Gerber is a full-stack developer with 15 years’ experience in the university sector, formerly a Technical Project Manager at The University of Queensland ITEE eResearchspecializing in Digital Humanities and Research Scientist at the Distributed System Technology Centre (DSTC). Anna is a JavaScript robotics enthusiast and maker who enjoys tinkering with soft circuits and 3D printers.

Asterisk has a wonderful, built-in ability to record calls. No additional software is required to make this happen. When Asterisk records a call, both sides of the call are recorded and written out to a file for playback on a computer. Call recording is often performed in call centers to ensure call quality, or to keep calls for later review, should the need arise. Asterisk provides the ability to record all of the calls, or to selectively record calls. In this article, we will look the following:   General recording options Recording calls to extensions Recording calls to queues Recording calls to conferences Maintaining call recordings Before enabling call recording for your PBX, make sure that you are aware of the legalities surrounding call recordings and privacy laws. Call recordings are prohibited in certain places, unless the caller is told that the call will be recorded. For example, in the state of California all of the parties on the call must consent to the call being recorded before it begins. Playing back a message stating that the call is being recorded prior to the call being answered is considered a valid form of consent. Recording formats FreePBX allows calls to be recorded in the following formats: WAV WAV49 ULAW ALAW SLN GSM Each format has its own ratio of file size to recording quality, and certain formats will not play on all of the computers. A comparison between all of the  available formats is as follows:

  HTML5 Multimedia Development Cookbook Recipes for practical, real-world HTML5 multimedia driven development.         Read more about this book       (For more resources on Multimedia development, see here.) Introduction "On the web, a man should not be judged by the color of his skin but by the content of his content." - Internet meme To be correct according to the specification semantically, we need to know what the content is so we can wrap it with the most appropriate new element tag. While this may mean we developers have to think differently, a new challenge is exactly why we're here. In this article we'll look at some examples of how to do just that using several of HTML5's new elements. "In case of conflict, consider users over authors over implementers over specifiers over theoretical purity." - Priority of Constituencies Throughout this article, we'll show you how to use the new <article> element to mark up both blog posts and comments, add a meaningful publication date to an <article>, use the new <mark> element to highlight text, and how to note visual elements using the new <figure> element. We'll then turn our attention to some new methods of styling text with font replacement techniques, as well as adding drop shadows and gradients to our text.   Structuring a blog article "The <article> element represents a self-contained composition in a document, page, application, or site and that is, in principle, independently distributable or reusable, e.g. in syndication. This could be a forum post, a magazine or newspaper article, a blog entry, a user-submitted comment, an interactive widget or gadget, or any other independent item of content." - WHATWG's HTML5 Draft Standard - http://whatwg.org/html5 Getting ready Blog entries are perfect candidates for the new <article> element, which is designed for syndicated content. For this recipe, let's start by identifying the major elements of a blog <article>: There's usually a headline in the form of a heading tag, the blog entry itself consisting of several paragraphs and perhaps one or more images, and some information that usually includes the author's name and other related metadata. Notice this is all self-contained related content. How to do it... We're going to continue using the new HTML5 <header> and <footer> elements. The headline, entry and meta-information should be wrapped in their own unique tags, like <h2>, multiple <p>s and the new &ltfooter>. Let's start with a foundation and add our new <article> element twice: <!DOCTYPE html><html lang="en"><head> <meta charset="UTF-8"> <title>Blog Title</title> <!--[if lt IE 9]><script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"> </script>[endif]--> <meta name="viewport" content="width=device-width, initial-scale=1.0"></head><body><article> <header> <h2>Headline</h2> </header> <p>First paragraph</p> <p>Second paragraph</p> <footer>Meta information.</footer></article><article> <header> <h2>Headline</h2> </header> <p>First paragraph</p> <p>Second paragraph</p> <footer>Meta information.</footer></article></body></html> Put your code on a diet? Ready for a shocker? Want to have your mind blown? The <html> and <head> and <body> tags (as well as their closing tags) are now optional in the HTML5 specification. Sure, you could leave them in there, and your pages will validate just fine, but why should we? If remove them from the previous code, we are left with the spartan: <!DOCTYPE html><meta charset="UTF-8"><title>Blog Title</title><!--[if lt IE 9]><script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"> </script>[endif]--><meta name="viewport" content="width=device-width, initial-scale=1.0"><article> <header> <h2>Headline</h2> </header> <p>First paragraph</p> <p>Second paragraph</p> <footer>Meta information.</footer></article><article> <header> <h2>Headline</h2> </header> <p>First paragraph</p> <p>Second paragraph</p> <footer>Meta information.</footer></article> Don't believe me? Run that code through the World Wide Web Consortium's validator at: http://validator.w3.org, and you'll see it displays correctly in the browser. Well, not so fast buster. The problem is that removing those elements breaks our code for screen readers. Uh oh. Strike one. Also, removing the <body> tag breaks our new HTML5-enabling JavaScript for Internet Explorer. Strike two. And guess what? You can see it coming, can't you? Yes, removing the <html> tag removes the language of the page. There it is: Strike three. So let's add those elements back in, shall we? <!DOCTYPE html><html lang="en"><head> <meta charset="UTF-8"> <title>Blog Title</title> <!--[if lt IE 9]><script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"> </script>[endif]--> <meta name="viewport" content="width=device-width, initial-scale=1.0"></head><body> <article> <header> <h2>Headline</h2> </header> <p>First paragraph</p> <p>Second paragraph</p> <footer>Meta information.</footer> </article> <article> <header> <h2>Headline</h2> </header> <p>First paragraph</p> <p>Second paragraph</p> <footer>Meta information.</footer> </article></body></html> There, that's better. How it works... Remember, the new <article> element is a collection of related information intended for syndication via RSS or another means. There's more... Richer, more meaningful semantics is perhaps the most significant goal for HTML5. It's better for machines, better for authors, and most importantly, better for our audiences. Validation as an aid, not a crutch As we saw previously, removing the &lthtml> and <head> and <body> tags render a still valid page. So that begs the question of how valid validators are. Unlike the XML world, HTML5 can use incorrect syntax and still render just fine. The author makes every effort to validate his code whenever possible. It's not necessary to be slavish to the validator, but it's always a good quality control check. And the closer you get to valid code, the better chance browsers will display your work in as consistent a manner as possible. Eric Meyer's funny The author loves how CSS guru Eric Meyer thinks about validators: Where to find validators You can make good use of code validators at: http://validator.nu http://validator.w3.org   Highlighting text using the mark element "The &ltmark> element represents a run of text in one document marked or highlighted for reference purposes, due to its relevance in another context. When used in a quotation or other block of text referred to from the prose, it indicates a highlight that was not originally present but which has been added to bring the reader's attention to a part of the text that might not have been considered important by the original author when the block was originally written, but which is now under previously unexpected scrutiny. When used in the main prose of a document, it indicates a part of the document that has been highlighted due to its likely relevance to the user's current activity." - WHATWG's HTML5 Draft Standard - http://whatwg.org/html5 Getting ready When viewing search results, you'll often find the term for which you searched highlighted. Instead of relying on a semantically meaningless tag, we can now use the more meaningful <mark> element. How to do it... In this recipe, you'll see HTML5doctor.com has an excellent example of how to use the new <mark> element to highlight a search results term. This gives a useful semantic hook not only for styling but also for the machine tracking the results. <!DOCTYPE html><html lang="en"><head> <meta charset="UTF-8"> <title></title> <!--[if lt IE 9]><script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"> </script>[endif]--> <meta name="viewport" content="width=device-width, initial-scale=1.0"></head><body> <h1>716,000,000 search results for the query "<mark>HTML 5</mark>"</h1> <section id="search-results"> <article> <h2><a href="http://en.wikipedia.org/wiki/HTML_5"> <mark>HTML 5</mark> - Wikipedia, the free encyclopedia</a></h2> <p><mark>HTML 5</mark> is the next major revision of <mark>HTML</mark> ("hypertext markup language"), the core markup language of the World Wide Web. The WHATWG started work on the ... <a href="http://en.wikipedia.org/wiki/HTML_5"> Read more</a></p> </article> <article> <h2><a href="http://dev.w3.org/html5/spec/Overview.html"> <mark>HTML 5</mark></a></h2> <p>A vocabulary and associated APIs for <mark>HTML</mark> and XHTML. Editor's Draft 16 August 2009. Latest Published Version: http://w3.org/TR/<mark>html5</mark>/; Latest Editor's ... <a href="http://dev.w3.org/html5/spec/Overview.html"> Read more</a></p> </article> </section></body></html> Adding a simple style declaration like: <style type="text/css"> mark {background-color: yellow; font-weight: bold;}</style> in the <head> section helps us render this highlighted text: How it works... The new <mark> element simply highlights a word or phrase to draw the reader's attention. To do this, simply specify the <mark> to be bold or italicized or highlighted in some way in your corresponding Cascading Style Sheet. There's more... Sure, you could mark up and style a search-results page to use the <b> or <i> or even <span> tags to indicate for which term the search took place, but each of those tags only affects the presentation layer. They lack meaning. The new &ltmark> element can accomplish the same visual effect, while also adding that extra meaning to your markup. In fact, the new &ltmark> element is full of win. <Mark> long and prosper Another great use of the new <mark> element is highlighting a date in a calendar picker, as we often see on any date-based reservation system website like Priceline.com. Priceline.com highlights the current date by default when booking your itinerary. Instead of using a semantically meaningless tag to achieve this, the new <mark> element could be a perfect candidate to use. Waiting for browsers The new <mark> element isn't fully supported by any web browser at the time of this writing. Though the extra semantic meaning may not be apparent to machine readers, we can still use the new <mark> element as a stylistic "hook" until the day its meaning is fully supported by a variety of browsers. Is "future proof" a word? Remember that HTML5's new elements attempt to add extra meaning to our markup. The goal is never to take away meaning or break pages. With this in mind, it becomes much more palatable to layer on new elements like the <mark> element that's not fully implemented by browsers yet. Even if its meaning is not fully understood by machines yet, it certainly does not hurt to add it and make our pages as "future proof" as we possibly can. See also In 2001, Carrie Bickner prepared the "New York Public Library Online Style Guide" (http://legacy.www.nypl.org/styleguide) for branches of the NYPL to use when updating their websites. In this seminal publication, Bickner made the case for web standards by separating content (markup) from presentation (Cascading Style Sheets) from behavior (JavaScript). The publication was extremely forward-thinking for the time and was in use for many years.   Using the time element "The &lttime> element represents either a time on a 24-hour clock, or a precise date in the proleptic Gregorian calendar, optionally with a time and a time-zone offset." - WHATWG's HTML5 Draft Standard - http://whatwg.org/html5 Getting ready The new <time> element is a powerful way to display time or a specific date. How to do it... In this recipe we'll display dates and times that will be readable for both humans and machines. Let's look at four examples. <!DOCTYPE html><html lang="en"><head> <meta charset="UTF-8"> <title></title> <!--[if lt IE 9]><script src=http://html5shiv.googlecode.com/svn/trunk/html5.js> </script>[endif]--> <meta name="viewport" content="width=device-width, initial-scale=1.0"></head><body> <article> <header> <h2>Headline</h2> <time datetime="2010-11-29">November 29, 2010</time> </header> <p>First paragraph</p> <p>Second paragraph</p> <footer>Meta information.</footer> </article> <article> <header> <h2>Headline</h2> <time datetime="2010-11-29">Nov. 29</time> </header> <p>First paragraph</p> <p>Second paragraph</p> <footer>Meta information.</footer> </article> <article> <header> <h2>Headline</h2> <time datetime="2010-11-29">the date this was written</time> </header> <p>First paragraph</p> <p>Second paragraph</p> <footer>Meta information.</footer> </article> <article> <header> <h2>Headline</h2> <time datetime="2010-11-29T11:34">the date and time this was written</time> </header> <p>First paragraph</p> <p>Second paragraph</p> <footer>Meta information.</footer> </article></body></html> How it works... We can use the new <time> element to indicate specific dates, times, or both. There's more... The new &lttime> element specifies an exact moment in time—not a time period. Odd rules One interesting aspect of the new <time> element is that you can't use a date before the Christian Era. You also can't use a date like "November 2010." Whatever date we specify must be a positive, specific date—not a relative one. The HTML5 Working Group continues to address this seemingly arbitrary restriction. <time>'s Time will come Browsers display the new <time> element but don't do anything special with it—yet. Always remember SEO Time. Why are we so obsessed with it? One very valid reason to focus on time and dates on the web is Search Engine Optimization. SEO, once seen as some sort of mysterious voodoo only black hatted wizards understood, is now everyone's responsibility online. You spend time creating good code and expect a writer to create content worth reading. Now go one step further and ensure that your intended audience can actually find the content you have taken the time to create. And the new <time> element is just one of the ways search engines draw attention to the most recent content. See also The new HTML5 <time> element is a possible addition to the Microformats movement. Microformats promise to add additional semantic meaning to our markup. Though not officially a standard, Microformats are slowly gaining acceptance in the web development community. Learn more at Microformats.org.  

Node Web Development A practical introduction to Node, the exciting new server-side JavaScript web development stack What's a module? Modules are the basic building block of constructing Node applications. We have already seen modules in action; every JavaScript file we use in Node is itself a module. It's time to see what they are and how they work. The following code to pull in the fs module, gives us access to its functions: var fs = require('fs'); The require function searches for modules, and loads the module definition into the Node runtime, making its functions available. The fs object (in this case) contains the code (and data) exported by the fs module. Let's look at a brief example of this before we start diving into the details. Ponder over this module, simple.js: var count = 0; exports.next = function() { return count++; } This defines an exported function and a local variable. Now let's use it: The object returned from require('./simple') is the same object, exports, we assigned a function to inside simple.js. Each call to s.next calls the function next in simple.js, which returns (and increments) the value of the count variable, explaining why s.next returns progressively bigger numbers. The rule is that, anything (functions, objects) assigned as a field of exports is exported from the module, and objects inside the module but not assigned to exports are not visible to any code outside the module. This is an example of encapsulation. Now that we've got a taste of modules, let's take a deeper look. Node modules Node's module implementation is strongly inspired by, but not identical to, the CommonJS module specification. The differences between them might only be important if you need to share code between Node and other CommonJS systems. A quick scan of the Modules/1.1.1 spec indicates that the differences are minor, and for our purposes it's enough to just get on with the task of learning to use Node without dwelling too long on the differences. How does Node resolve require('module')? In Node, modules are stored in files, one module per file. There are several ways to specify module names, and several ways to organize the deployment of modules in the file system. It's quite flexible, especially when used with npm, the de-facto standard package manager for Node. Module identifiers and path names Generally speaking the module name is a path name, but with the file extension removed. That is, when we write require('./simple'), Node knows to add .js to the file name and load in simple.js. Modules whose file names end in .js are of course expected to be written in JavaScript. Node also supports binary code native libraries as Node modules. In this case the file name extension to use is .node. It's outside our scope to discuss implementation of a native code Node module, but this gives you enough knowledge to recognize them when you come across them. Some Node modules are not files in the file system, but are baked into the Node executable. These are the Core modules, the ones documented on nodejs.org. Their original existence is as files in the Node source tree but the build process compiles them into the binary Node executable. There are three types of module identifiers: relative, absolute, and top-level. Relative module identifiers begin with "./" or "../" and absolute identifiers begin with "/". These are identical with POSIX file system semantics with path names being relative to the file being executed. Absolute module identifiers obviously are relative to the root of the file system. Top-level module identifiers do not begin with "." , "..", or "/" and instead are simply the module name. These modules are stored in one of several directories, such as a node_modules directory, or those directories listed in the array require.paths, designated by Node to hold these modules. Local modules within your application The universe of all possible modules is split neatly into two kinds, those modules that are part of a specific application, and those modules that aren't. Hopefully the modules that aren't part of a specific application were written to serve a generalized purpose. Let's begin with implementation of modules used within your application. Typically your application will have a directory structure of module files sitting next to each other in the source control system, and then deployed to servers. These modules will know the relative path to their sibling modules within the application, and should use that knowledge to refer to each other using relative module identifiers. For example, to help us understand this, let's look at the structure of an existing Node package, the Express web application framework. It includes several modules structured in a hierarchy that the Express developers found to be useful. You can imagine creating a similar hierarchy for applications reaching a certain level of complexity, subdividing the application into chunks larger than a module but smaller than an application. Unfortunately there isn't a word to describe this, in Node, so we're left with a clumsy phrase like "subdivide into chunks larger than a module". Each subdivided chunk would be implemented as a directory with a few modules in it. In this example, the most likely relative module reference is to utils.js. Depending on the source file which wants to use utils.js it would use one of the following require statements: var utils = require('./lib/utils'); var utils = require('./utils'); var utils = require('../utils');  

Lets get started. Moodle's PreDefined Roles Moodle comes with a number of predefined roles. These standard roles are suitable for some educational setups, but most institutions require modifications to the roles' system in order to tailor Moodle to their specific needs. Each role has permissions for a number of actions that can be carried out in Moodle. For example, an administrator and a course creator are able to create new courses, whereas all other roles are denied this right. Likewise, a teacher is allowed to moderate forums, whereas students are only allowed to contribute to them. Before we can actually do anything with roles, we need to understand the concept of contexts, which is dealt with next. Contexts Contexts are the areas in Moodle where roles can be assigned to users. A role can be assigned within different contexts. A user has a role in any given context, where a context can be a course, an activity module, a user, a block, or Moodle itself. Moodle comes with the following seven contexts that you will come across a lot in this article.

(For more resources related to this topic, see here.) Basic file structure of a WordPress theme As WordPress developers, you should have a fairly good idea about the default file structure of WordPress themes. Let's have a brief introduction of the default files before identifying their usage in web applications. Think about a typical web application layout where we have a common header, footer, and content area. In WordPress, the content area is mainly populated by pages or posts. The design and the content for pages are provided through the page.php template, while the content for posts is provided through one of the following templates: index.php archive.php category.php single.php Basically, most of these post-related file types are developed to cater to the typical functionality in blogging systems, and hence can be omitted in the context of web applications. Since custom posts are widely used in application development, we need more focus on templates such as single-{post_type} and archive-{post_type} than category.php, archive.php, and tag.php. Even though default themes contain a number of files for providing default features, only the style.css and index.php files are enough to implement a WordPress theme. Complex web application themes are possible with the standalone index.php file. In normal circumstances, WordPress sites have a blog built on posts, and all the remaining content of the site is provided through pages. When referring to pages, the first thing that comes to our mind is the static content. But WordPress is a fully functional CMS, and hence the page content can be highly dynamic. Therefore, we can provide complex application screens by using various techniques on pages. Let's continue our exploration by understanding the theme file execution hierarchy. Understanding template execution hierarchy WordPress has quite an extensive template execution hierarchy compared to general web application frameworks. However, most of these templates will be of minor importance in the context of web applications. Here, we are going to illustrate the important template files in the context of web applications. The complete template execution hierarchy can be found at: http://hub.packtpub.com/wp-content/uploads/2013/11/Template_Hierarchy.png An example of the template execution hierarchy is as shown in the following diagram: Once the Initial Request is made, WordPress looks for one of the main starting templates as illustrated in the preceding screenshot. It's obvious that most of the starting templates such as front page, comments popup, and index pages are specifically designed for content management systems. In the context of web applications, we need to put more focus into both singular and archive pages, as most of the functionality depends on top of those templates. Let's identify the functionality of the main template files in the context of web applications: Archive pages: These are used to provide summarized listings of data as a grid. Single posts: These are used to provide detailed information about existing data in the system. Singular pages: These are used for any type of dynamic content associated with the application. Generally, we can use pages for form submissions, dynamic data display, and custom layouts. Let's dig deeper into the template execution hierarchy on the Singular Page path as illustrated in the following diagram: Singular Page is divided into two paths that contain posts or pages. Static Page is defined as Custom or Default page templates. In general, we use Default page templates for loading website pages. WordPress looks for a page with the slug or ID before executing the default page.php file. In most scenarios, web application layouts will take the other route of Custom page templates where we create a unique template file inside the theme for each of the layouts and define it as a page template using code comments. We can create a new custom page template by creating a new PHP file inside the theme folder and using the Template Name definition in code comments illustrated as follows: <?php/** Template Name: My Custom Template*/?> To the right of the preceding diagram, we have Single Post Page, which is divided into three paths called Blog Post, Custom Post, and Attachment Post. Both Attachment Posts and Blog Posts are designed for blogs and hence will not be used frequently in web applications. However, the Custom Post template will have a major impact on application layouts. As with Static Page, Custom Post looks for specific post type templates before looking for a default single.php file. The execution hierarchy of an Archive Page is similar in nature to posts, as it looks for post-specific archive pages before reverting to the default archive.php file. Now we have had a brief introduction to the template loading process used by WordPress. In the next section, we are going to look at the template loading process of a typical web development framework to identify the differences. Template execution process of web application frameworks Most stable web application frameworks use a flat and straightforward template execution process compared to the extensive process used by WordPress. These frameworks don't come with built-in templates, and hence each and every template will be generated from scratch. Consider the following diagram of a typical template execution process: In this process, Initial Request always comes to the index.php file, which is similar to the process used by WordPress or any other framework. It then looks for custom routes defined within the framework. It's possible to use custom routes within a WordPress context, even though it's not used generally for websites or blogs. Finally, Initial Request looks for the direct template file located in the templates section of the framework. As you can see, the process of a normal framework has very limited depth and specialized templates. Keep in mind that index.php referred to in the preceding section is the file used as the main starting point of the application, not the template file. In WordPress, we have a specific template file named index.php located inside the themes folder as well. Managing templates in a typical application framework is a relatively easy task when compared to the extensive template hierarchy used by WordPress. In web applications, it's ideal to keep the template hierarchy as flat as possible with specific templates targeted towards each and every screen. In general, WordPress developers tend to add custom functionalities and features by using specific templates within the hierarchy. Having multiple templates for a single screen and identifying the order of execution can be a difficult task in large-scale applications, and hence should be avoided in every possible instance. Web application layout creation techniques As we move into developing web applications, the logic and screens will become complex, resulting in the need of custom templates beyond the conventional ones. There is a wide range of techniques for putting such functionality into the WordPress code. Each of these techniques have their own pros and cons. Choosing the appropriate technique is vital in avoiding potential bottlenecks in large-scale applications. Here is a list of techniques for creating dynamic content within WordPress applications: Static pages with shortcodes Page templates Custom templates with custom routing Summary In this article we learned about basic file structure of the WordPress theme, the template execution hierarchy, and template execution process. We also learned the different techniques of Web application layout creation. Resources for Article: Further resources on this subject: Customizing WordPress Settings for SEO [Article] Getting Started with WordPress 3 [Article] Dynamic Menus in WordPress [Article]

General attributes These methods get and set DOM attributes of elements. .attr() (getter) Get the value of an attribute for the first element in the set of matched elements. .attr(attributeName) Parameters attributeName: The name of the attribute to get Return value A string containing the attribute value. Description It's important to note that the .attr() method gets the attribute value for only the first element in the matched set. To get the value for each element individually, we need to rely on a looping construct such as jQuery's .each() method. Using jQuery's .attr() method to get the value of an element's attribute has two main benefits: Convenience: It can be called directly on a jQuery object and chained to other jQuery methods. Cross-browser consistency: Some attributes have inconsistent naming from browser to browser. Furthermore, the values of some attributes are reported inconsistently across browsers, and even across versions of a single browser. The .attr() method reduces such inconsistencies. .attr() (setter) Set one or more attributes for the set of matched elements. .attr(attributeName, value).attr(map).attr(attributeName, function) Parameters (first version) attributeName: The name of the attribute to set value: A value to set for the attribute Parameters (second version) map: A map of attribute-value pairs to set Parameters (third version) attributeName: The name of the attribute to set function: A function returning the value to set Return value The jQuery object, for chaining purposes. Description The .attr() method is a convenient and powerful way to set the value of attributes, especially when setting multiple attributes or using values returned by a function. Let's consider the following image: <img id="greatphoto" src="brush-seller.jpg" alt="brush seller" /> Setting a simple attribute We can change the alt attribute by simply passing the name of the attribute and its new value to the .attr() method. $('#greatphoto').attr('alt', 'Beijing Brush Seller'); We can add an attribute the same way. $('#greatphoto').attr('title', 'Photo by Kelly Clark'); Setting several attributes at once To change the alt attribute and add the title attribute at the same time, we can pass both sets of names and values into the method at once using a map (JavaScript object literal). Each key-value pair in the map adds or modifies an attribute: $('#greatphoto').attr({alt: 'Beijing Brush Seller',title: 'photo by Kelly Clark'}); When setting multiple attributes, the quotation marks around attribute names are optional. Computed attribute values By using a function to set attributes, we can compute the value based on other properties of the element. For example, we could concatenate a new value with an existing value as follows: $('#greatphoto').attr('title', function() { return this.alt + ' – photo by Kelly Clark'}); This use of a function to compute attribute values can be particularly useful when we modify the attributes of multiple elements at once. .removeAttr() Remove an attribute from each element in the set of matched elements. .removeAttr(attributeName).removeAttr(function) Parameters (first version) attributeName: An attribute to remove Parameters (second version) function: A function returning the attribute to remove Return value The jQuery object, for chaining purposes. Description The .removeAttr() method uses the JavaScript removeAttribute() function, but it has the advantage of being able to be called directly on a jQuery object and it accounts for different attribute naming across browsers. As of jQuery 1.4, the .removeAttr() function allows us to indicate the attribute to be removed by passing in a function.

(For more resources related to this topic, see here.) Getting ready Any learner taking a quiz may want to know how well he/she has answered the questions posed. Often, working with Moodle, the instructor is at a distance from the learner. Providing feedback is a great way of enhancing communication between learner and instructor. Learner feedback can be provided at multiple levels using Moodle Quiz. You can create feedback at various levels in both the questions and the overall quiz. Here we will examine feedback at the question level. General feedback When we add General Feedback to a question, every student sees the feedback, regardless of their answer to the question. This is good opportunity to provide clarification for the learner who had guessed a correct answer, as well as for the learner whose response was incorrect. Individual response feedback We can create feedback tailored to each possible response in a multiple choice question. This feedback can be more focused in nature. Often, a carefully crafted distracter in a multiple choice can reveal misconceptions and the feedback can provide the correction required as soon as the learner completes the quiz. Feedback given when the question is fresh in the learner's mind, is very effective. How to do it... Let's create some learner feedback for some of the questions that we have created in the question bank: First of all, let's add general feedback to a question. Returning to our True-False question on Texture, we can see that general feedback is effective when there are only two choices. Remember that this type of feedback will appear for all learners, regardless of the answer they submitted. The intention of this feedback is to reflect the correct solution and also give more background information to enhance the teaching opportunity. Let's take a look at how to create a specific feedback for each possible response that a learner may submit. This is done by adding individual response feedback. Returning to our multiple choice question on application of the element line, a specific feedback response tailored to each possible choice will provide helpful clarification for the student. This type of feedback is entered after each possible choice. Here is an example of a feedback to reinforce a correct response and a feedback for an incorrect response: In this case, the feedback the learner receives is tailored to the response they have submitted. This provides much more specific feedback to the learner's choice of responses. For the embedded question (Cloze), feedback is easy to add in Moodle 2.0. In the following screenshot, we can see the question that we created with feedback added: And this is what the feedback looks like to the student: How it works... We have now improved questions in our exam bank by providing feedback for the learner. We have created both general feedback that all learners will see and specific feedback for each response the learner may choose. As we think about the learning experience for the learner, we can see that immediate feedback with our questions is an effective way to reinforce learning. This is another feature that makes Moodle Quiz such a powerful tool. There's more... As we think about the type of feedback we want for the learner, we can combine feedback for individual responses with general feedback. Also there are options for feedback for any correct response, for any partially correct response, or for any incorrect response. Feedback serves to engage the learners and personalize the experience. We created question categories, organized our questions into categories, and learned how to add learner feedback at various levels inside the questions. We are now ready to configure a quiz. Summary In the article we have seen how we can add feedback to the questions of the Moodle Quiz. Resources for Article : Further resources on this subject: Integrating Moodle 2.0 with Mahara and GoogleDocs for Business [Article] What's New in Moodle 2.0 [Article] Moodle 2.0 FAQs [Article]

jQuery UI Themes Beginner's Guide Create new themes for your JQuery site with this step-by-step guide ThemeRoller basics Before we start using the ThemeRoller application to design and build our own themes, we'll take a quick look at what makes it such a handy tool. There is a lot more to the ThemeRoller than simply changing themes—we also use it to build them. You can think of it as an IDE for jQuery UI themes. Instant feedback What makes the ThemeRoller application such a powerful development tool is the speed with which you get feedback to changes made in the theme design. Any change made in the ThemeRoller is instantaneously reflected in the sample widgets provided on the page. For instance, if I were to change a font setting, that change would be reflected immediately in the sample widgets provided on the same page. There is no need to update the application you're building to see the results of small adjustments made to theme style settings. The same is true of prepackaged themes in the ThemeRoller gallery. Selecting a theme will apply it to the same widgets - you get immediate feedback. This is very helpful in deciding on prepackaged themes. If you can see how it looks with jQuery UI widgets right away, that may dissuade you from using the theme or it may close the deal. The idea behind this feedback mechanism offered by the ThemeRoller is a sped-up development cycle. Eliminating several steps when developing anything, themes included, is a welcome feature. The dev tool The ThemeRoller dev tool is a simple bookmarket for Firefox that brings the entire ThemeRoller application into any page with jQuery UI widgets. The benefit of the dev tool is that it allows you to see immediate theme changes in the context of the application you're building. If you use the ThemeRoller application from the jQuery UI website, you can only see changes as they apply to the sample widgets provided. This can give you a better idea of what the theme changes will look like on a finished product. There are some limitations to using the dev tool though. If you're developing your application locally, not on a development server, you can't use the dev tool due to security restrictions. The dev tool is better suited for viewing changes to themes, or viewing different themes entirely, on a deployed user interface. Having said that, if you're designing a user interface with several collaborators, you might have a remote development server. In this scenario, the dev tool suits its name. Portability The ThemeRoller application is portable in more ways than one. The dev tool for Firefox allows us to use the application within any jQuery UI application. This means that we can design and tweak our jQuery UI themes as we build the widgets. This portability between applications means that we can build a single theme that works for a suite of applications, or a product line, if we're so inclined. We can also use the ThemeRoller application directly from the jQueryUI website. This is handy if we don't have any widgets built or if you're trying jQuery UI out for the first time and just want to browse the wide selection of prepackaged themes. Whatever approach you take, the application is the same and will always be consistent, as it is a hosted application. You don't need to concern yourself with installing an IDE for theme authors to collaborate with. The ThemeRoller application is available wherever they are. ThemeRoller gallery It is nice to have a wide variety of prepackaged themes to choose from. It isn't all that helpful if you can't see how they look. The ThemeRoller application has a gallery where we can not only browse prepackaged themes but also take them for a test drive. This section is about using the ThemeRoller gallery to view themes and get a feel of the variety available to us. Viewing themes The ThemeRoller application doesn't hide anything about the prepackaged themes in the gallery. When we preview a theme, we get to see how it looks when applied to widgets. The theme gallery even gives us a thumbnail in the browser to show a bird's eye view of the theme. So if you see a lot of black and you're looking for something bright, you don't need to bother selecting it to see how the widgets look with it. Time for action - previewing a theme It's time for us to preview a jQuery UI theme before we actually download it. We can get an idea of what a theme in the ThemeRoller gallery will look like when applied to widgets: Point your web browser to http://jqueryui.com/themeroller/. Select the Gallery tab in the ThemeRoller section on the right-hand side. Move your mouse pointer over any theme in the gallery. A visual indicator will be displayed. Select the theme thumbnail:   What just happened? We've just selected a theme to preview from the ThemeRoller gallery. You'll notice that all the sample widgets to the right are instantly updated with the new theme. If we wanted to, we could change our theme selection and the sample widgets are once again updated with the theme changes. You'll notice that once you make a theme selection, the URL in your address bar is now long and ugly. These are the individual theme settings for the chosen theme being passed to the ThemeRoller page with the sample widgets. You'll also notice that the theme selection on the left-hand side of the page isn't preserved. This is because we're passing individual theme settings and not the name of the theme itself, for example, instancetheme=darkness. We'll see why this distinction is important in a little bit. Downloading themes Once you've selected a theme from the gallery and you're happy with how it looks, it is time to download it and use it with your jQuery UI project. Downloading a theme is easy—each prepackaged theme has a download button that will take you to the jQuery UI download page. If we wanted to, we could download all themes in a single package to experiment with, locally. This would also eliminate the need for the ThemeRoller application, which you probably don't want to do. Time for action - downloading a theme The gallery is a nice way to preview a theme, but now we want to use it in our application. To do this, we need to download it. This is similar to downloading the jQuery UI toolkit: Point your web browser to http://jqueryui.com/themeroller/. Select the Gallery tab in the ThemeRoller section on the left-hand side. Find a theme you wish to download. Click on the Download button underneath the theme thumbnail. This will bring you to the jQuery UI download page. Notice that your chosen theme is selected on the right-hand side of the page. Click on the Download button to download your theme: What just happened? We've just selected a prepackaged theme from the ThemeRoller gallery and downloaded it. In fact, you just downloaded jQuery UI again. The difference being, the downloaded ZIP archive contains the theme you selected from the gallery. The same principles apply for extracting the archive and using your theme with your jQuery UI application. The downside is that if you're downloading a theme, chances are you already have a jQuery UI application under development. In this case, downloading jQuery UI JavaScript files is redundant. However, there is no easy way around this. This is one of the drawbacks to having a useful tool available to us—a minor drawback at that. If you're only interested in the theme, you simply need to extract the theme folder from the ZIP archive and copy it to your jQuery UI application directory. You then need to update your path in your HTML in including the appropriate CSS file. You'll also notice that after clicking on the Download button from the theme gallery, you're brought to the download page with an ugly URL. That is, you'll see something like /download/?themeParams=%3FffDefault instead of just /download. This is a requirement of the ThemeRoller application that allows developers to edit existing themes or to roll their own. Without these parameters, we wouldn't be able to download themes we have made changes to. The jQuery UI download page also includes an Advanced Settings section that is hidden by default. This is because you rarely need to use it. It allows you to set the CSS scope for your theme, useful if you're using multiple themes in a single user interface. This isn't a recommended practice; the key idea behind jQuery UI themes is consistency. The advanced settings also lets you change the name of the downloaded theme folder. This can be useful if you plan on changing your theme later, but you can always rename the folder after downloading it.

(For more resources on Joomla!, see here.) Building a basic Joomla! module is not that difficult. In fact it's quite easy. Just stay with me, and we will divide the task into some easy-to-follow steps. First of all, we need to create a folder for our module, for example, mod_littlecontact. This folder is where we will place all the files necessary for our module. For example, one of the files we are going to need is the mod_littlecontact.php file, which is named exactly the same as the folder, but with a .php extension. Let's see what we need to put in it: <?phpdefined('_JEXEC') or die('Direct Access to this location is notallowed.');?><h1>Just a simple contact form!</h1> We will look at just the basics. First, defined('_JEXEC') checks whether the file has been included by Joomla! instead of being called directly. If it has been included by Joomla!, the _JEXEC constant would have been defined. With this PHP file created we need to create another file, an XML one this time. We will call it mod_littlecontact.xml; notice that, again, the name is the same as the folder one. Just create the file, and after that we will place the following contents in it: <?xml version="1.0" encoding="utf-8"?><install type="module" version="1.5.0"> <name>Little Contact Form</name> <author>Jose Argudo Blanco</author> <creationDate>2010</creationDate> <copyright>All rights reserved by Jose Argudo Blanco.</copyright> <license>GPL 2.0</license> <authorEmail>jose@joseargudo.com</authorEmail> <authorUrl>www.joseargudo.com</authorUrl> <version>1.0.0</version> <description>A simple contact form</description> <files> <filename module="mod_littlecontact"> mod_littlecontact.php</filename> </files></install> Most of the contents of this XML file are quite easy to follow and very self-explanatory. In the files section, we have included all the files necessary for our module. Notice that we do not include the XML file itself. With these two files created, we can give a try to this simple module. Copying this folder into our Joomla! modules folder won't work, as Joomla! requires us to install the extension through the Extensions | Install/Uninstall menu. So, what do we need to do? Just compress these two files into a ZIP file by using any tool of your liking. At the end we will need to have a mod_littlecontact.zip file with the following two files inside: mod_littlecontact.php mod_littlecontact.xml Installing our module is done exactly as with any other modules. Go to the administrator screen of our site, then go to the Extensions | Install/Uninstall menu, search and select the file, and then click on Upload File & Install button. If all goes OK, and it really should, we will be able to find our module listed in Extensions | Module Manager, as seen in the following screenshot: We can click in the module name, just as we would do with any of the others. If we enter the administration panel of the module we will see a screen very much like the other modules, as Joomla! standardizes this screen. Just take a look at the Details zone, which will look like the next screenshot: He re we can select the parameters we want, and enable the module. This time we will place it in the module_7 position of our template. Also note that the description is the one that we place in the module XML file: <description>A simple contact form</description> After we have enabled the module, we will be able to see it in the frontend, in the module position we have selected: There's not too much for now, but it's working! Now we will enhance it and convert it into a contact form. Note that now that we have installed our module, a new folder will have been created into our Joomla! installation. We can find this folder in the modules folder, it will be called mod_littlecontact. So now we have this structure on our Joomla! Site: modules/ mod_littlecontact/ mod_littlecontact.php mod_littlecontact.xml As the module is already installed, we can modify these files, and we will be able to see the changes without needing to reinstall it. We have just accomplished our first step; the basics are there, and now we can concentrate on making our modifications. Creating a "Send us a question" module One of the first things we are going to create is an empty index.html file; this will be used so that no one can take a look at the folder structure for the module. For example, imagine that our site is installed in http://wayofthewebninja.com. If we go to http://wayofthewebninja.com/modules/mod_littlecontact/ we will see something like the next image: If we try to click on mod_littlecontact.php, we will see the following phrase: Direct Access to this location is not allowed. That's because the code we added to our file is as follows: <?phpdefined('_JEXEC') or die('Direct Access to this location is notallowed.');?> Of course, we don't want people to be able to see which files we are using for our module. For this place, we used the empty index.html file mentioned in the modules/mod_littlecontact folder. This way, if anyone tries to go to http://wayofthewebninja.com/modules/mod_ littlecontact/, they will see only an empty screen. Good, now note that when we add any file, we need to reflect it on the mod_littlecontact.xml file in the files section: <files> <filename module="mod_littlecontact">mod_littlecontact.php</filename> <filename>index.html</filename></files> This way, when we pack the file for install, the installation process will take this file into account, otherwise it will be left out. Once we have done this, we are going to create another file, a CSS one this time, so we can put our styles in it. For this we are going to first create a new folder, also called css. It will be placed in modules/mod_littlecontact/. Inside that folder we will create a file called styles.css; this file also needs to be declared in the XML: <filename>css/styles.css</filename> In this modules/mod_littlecontact/css/styles.css file we are going to place the following code: #littlecontact h1{ font-size: 18px; border-bottom: 1px solid #ffffff;} But then, if we are to apply these styles, we need to load this CSS file. How are we going to do this? Open the modules/mod_littlecontact/mod_littlecontact.php file and modify it as follows: <?phpdefined('_JEXEC') or die('Direct Access to this location is notallowed.');JHTML::stylesheet('styles.css','modules/mod_littlecontact/css/');?><div id="littlecontact"> <h1>Just a simple contact form!</h1></div> There's not much change here; we have enveloped our previous content in a DIV, with the littlecontact ID, so that we can target our styles. This is the easy part, but there's also an important one, shown as follows: JHTML::stylesheet('styles.css','modules/mod_littlecontact/css/'); We are using the JHTML::stylesheet method to create a link, in our header section, to our CSS file. In fact, if we check the source code on our frontend, we will see: <link rel="stylesheet" href="/modules/mod_littlecontact/css/styles.css" type="text/css" /> This way our stylesheet will be loaded, and our module will look like the next screenshot: As we can see, our styles have been applied. The JHTML::stylesheet method is quite easy to use, the first parameter being the file and the second one being the path to the file. Now we are going to prepare our simple form. Again we will modify our mod_littlecontact.php file, and now it will look more like the following: <?phpdefined('_JEXEC') or die('Direct Access to this location is notallowed.');JHTML::stylesheet('styles.css','modules/mod_littlecontact/css/');?><div id="littlecontact"> <h1>Just a simple contact form!</h1> <form action="index.php" method="post" id="sc_form"> <label>Your name:</label><br/> <input type="text" name="your_name" value="" size="40" class="sc_input"/><br/><br/> <label>Your question:</label><br/> <textarea name="your_question" class="sc_input" rows="5" cols="30"></textarea><br/><br/> <input type="submit" name="send" value="Send" class="sc_button" /> </form></div> This is a common HTML form. We need some styling here, just to make it look good. Let's make the following minimal changes to our styles.css file: #littlecontact h1{ font-size: 18px; border-bottom: 1px solid #ffffff; margin-bottom: 15px;}.sc_input{ border: 1px solid #3A362F;}.sc_button{ background-color: #3A362F; border: 0; color: #ffffff; padding: 5px;} Most styles are new, and modifications to previous h1 styling have been marked. With this minimal change our module looks a bit better. You can see it in the following screenshot:

(For more resources on PHP, see here.) The problem Dealing with sessions can be confusing, and is also a source of security loopholes. So we want our CMS framework to provide basic mechanisms that are robust. We want them to be easy to use by more application-oriented software. To achieve these aims, we need to consider: The need for sessions and their working The pitfalls that can introduce vulnerabilities Efficiency and scalability considerations Discussion and considerations To see what is required for our session handling, we shall first review the need for them and consider how they work in a PHP environment. Then the vulnerabilities that can arise through session handling will be considered. Web crawlers for search engines and more nefarious activities can place a heavy and unnecessary load on session handling, so we shall look at ways to avoid this load. Finally, the question of how best to store session data is studied. Why sessions? The need for continuity was mentioned when we first discussed users. But it is worth reviewing the requirement in a little more detail. If Tim Berners-Lee and his colleagues had known all the developments that would eventually occur in the internet world, maybe the Web would have been designed differently. In particular, the basic web transport protocol HTTP might not have treated each request in isolation. But that is hindsight, and the Web was originally designed to present information in a computer-independent way. Simple password schemes were sufficient to control access to specific pages. Nowadays, we need to cater for complex user management, or to handle things like shopping carts, and for these we need continuity. Many people have recognized this, and introduced the idea of sessions. The basic idea is that a session is a series of requests from an individual website visitor, and the session provides access to enduring information that is available throughout the session. The shopping cart is an obvious example of information being retained across the requests that make up a session. PHP has its own implementation of sessions, and there is no point reinventing the wheel, so PHP sessions are the obvious tool for us to use to provide continuity. How sessions work There are three main choices which have been available for handling continuity: Adding extra information to the URI Using cookies Using hidden fields in the form sent to the browser All of them can be used at times. Which of them is most suitable for handling sessions? PHP uses either of the first two alternatives. Web software often makes use of hidden variables, but they do not offer a neat way to provide an unobtrusive general mechanism for maintaining continuity. In fact, whenever hidden variables are used, it is worth considering whether session data would be a better alternative. For reasons discussed in detail later, we shall consider only the use of cookies, and reject the URI alternative. There was a time when there were lots of scary stories about cookies, and people were inclined to block them. While there will always be security issues associated with web browsing, the situation has changed, and the majority of sites now rely on cookies. It is generally considered acceptable for a site to demand the use of cookies for operations such as user login or for shopping carts and purchase checkout. The PHP cookie-based session mechanism can seem obscure, so it is worth explaining how it works. First we need to review the working of cookies. A cookie is simply a named piece of data, usually limited to around 4,000 bytes, which is stored by the browser in order to help the web server to retain information about a user. More strictly, the connection is with the browser, not the user. Any cookie is tied to a specific website, and optionally to a particular part of the website, indicated by a path. It also has a life time that can be specified explicitly as a duration; a zero duration means that the cookie will be kept for as long as the browser is kept open, and then discarded. The browser does nothing with cookies, except to save and then return them to the server along with requests. Every cookie that relates to the particular website will be sent if either the cookie is for the site as a whole, or the optional path matches the path to which the request is being sent. So cookies are entirely the responsibility of the server, but the browser helps by storing and returning them. Note that, since the cookies are only ever sent back to the site that originated them, there are constraints on access to information about other sites that were visited using the same browser. In a PHP program, cookies can be written by calling the set_cookie function, or implicitly through session handling. The name of the cookie is a string, and the value to be stored is also a string, although the serialize function can be used to make more structured data into a string for storage as a cookie. Take care to keep the cookies within the size limit. PHP makes available the cookies that have been sent back by the browser in the $_COOKIES super-global, keyed by their names. Apart from any cookies explicitly written by code, PHP may also write a session cookie. It will do so either as a result of calls to session handling functions, or because the system has been configured to automatically start or resume a session for each request. By default, session cookies do not use the option of setting an expiry time, but can be deleted when the browser is closed down. Commonly, browsers keep this type of cookie in memory so that they are automatically lost on shutdown. Before looking at what PHP is doing with the session cookie, let's note that there is an important general consideration for writing cookies. In the construction of messages between the server and the browser, cookies are part of the header. That means rules about headers must be obeyed. Headers must be sent before anything else, and once anything else has been sent, it is not permitted to send more headers. So, in the case of server to browser communication, the moment any part of the XHTML has been written by the PHP program, it is too late to send a header, and therefore too late to write a cookie. For this reason, a PHP session is best started early in the processing. The only purpose PHP has in writing a session cookie is to allocate a unique key to the session, and retrieve it again on the next request. So the session cookie is given an identifying name, and its value is the session's unique key. The session key is usually called the session ID, and is used by PHP to pick out the correct set of persistent values that belong to the session. By default, the session name is PHPSESSID but it can, in most circumstances, be changed by calling the PHP function session_name prior to starting the session. Starting, or more often restarting, a session is done by calling session_start, which returns the session ID. In a simple situation, you do not need the session ID, as PHP places any existing session data in another superglobal, $_SESSION. In fact, we will have a use for the session ID as you will soon see. The $_SESSION super-global is available once session_start has been called, and the PHP program can store whatever data it chooses in it. It is an array, initially empty, and naturally the subscripts need to be chosen carefully in a complex system to avoid any clashes. The neat part of the PHP session is that provided it is restarted each time with session_start, the $_SESSION superglobal will retain any values assigned during the handling of previous requests. The data is thus preserved until the program decides to remove it. The only exception to this would be if the session expired, but in a default configuration, sessions do not expire automatically. Later in this article, we will look at ways to deliberately kill sessions after a determinate period of inactivity. As it is only the session ID that is stored in the cookie, rules about the timing of output do not apply to $_SESSION, which can be read or written at any time after session_start has been called. PHP stores the contents of $_SESSION at the end of processing or on request using the PHP function session_write_close. By default, PHP puts the data in a temporary file whose name includes the session ID. Whenever the session data is stored, PHP retrieves it again at the next session_start. Session data does not have to be stored in temporary files, and PHP permits the program to provide its own handling routines. We will look at a scheme for storing the session data in a database later in the article. Avoiding session vulnerabilities So far, the option to pass the session ID as part of the URI instead of as a cookie has not been considered. Looking at security will show why. The main security issue with sessions is that a cracker may find out the session ID for a user, and then hijack that user's session. Session handling should do its best to guard against that happening. PHP can pass the session ID as part of the URI. This makes it especially vulnerable to disclosure, since URIs can be stored in all kinds of places that may not be as inaccessible as we would like. As a result, secure systems avoid the URI option. It is also undesirable to find links appearing in search engines that include a session ID as part of the URI. These two points are enough to rule out the URI option for passing session ID. It can be prevented by the following PHP calls: ini_set('session.use_cookies', 1);ini_set('session.use_only_cookies', 1); These calls force PHP to use cookies for session handling, an option that is now considered acceptable. The extent to which the site will function without cookies depends on what a visitor can do with no continuity of data—user login will not stick, and anything like a shopping cart will not be remembered. It is best to avoid the default name of PHPSESSID for the session cookie, since that is something that a cracker could look for in the network traffic. One step that can be taken is to create a session name that is the MD5 hash of various items of internal information. This makes it harder but not impossible to sniff messages to find out a session ID, since it is no longer obvious what to seek—the well known name of PHPSESSID is not used. It is important for the session ID to be unpredictable, but we rely on PHP to achieve that. It is also desirable that the ID be long, since otherwise it might be possible for an attacker to try out all possible values within the life of a session. PHP uses 32 hexadecimal digits, which is a reasonable defense for most purposes. The other main vulnerability apart from session hijacking is called session fixation. This is typically implemented by a cracker setting up a link that takes the user to your site with a session already established, and known to the cracker. An important security step that is employed by robust systems is to change the session ID at significant points. So, although a session may be created as soon as a visitor arrives at the site, the session ID is changed at login. This technique is used by Amazon among others so that people can browse for items and build up a shopping cart, but on purchase a fresh login is required. Doing this reduces the available window for a cracker to obtain, and use, the session ID. It also blocks session fixation, since the original session is abandoned at critical points. It is also advisable to change the ID on logout, so although the session is continued, its data is lost and the ID is not the same. It is highly desirable to provide logout as an option, but this needs to be supplemented by time limits on inactive sessions. A significant part of session handling is devoted to keeping enough information to be able to expire sessions that have not been used for some time. It also makes sense to revoke a session that seems to have been used for any suspicious activity. Ideally, the session ID is never transmitted unencrypted, but achieving this requires the use of SSL, and is not always practical. It should certainly be considered for high security applications. Search engine bots One aspect of website building is, perhaps unexpectedly, the importance of handling the bots that crawl the web. They are often gathering data for search engines, although some have more dubious goals, such as trawling for e-mail addresses to add to spam lists. The load they place on a site can be substantial. Sometimes, search engines account for half or more of the bandwidth being used by a site, which certainly seems excessive. If no action is taken, these bots can consume significant resources, often for very little advantage to the site owner. They can also distort information about the site, such as when the number of current visitors is displayed but includes bots in the counts. Matters are made worse by the fact that bots will normally fail to handle cookies. After all, they are not browsers and have no need to implement support for cookies. This means that every request by a bot is separate from every other, as our standard mechanism for linking requests together will not work. If the system starts a new session, it will have to do this for every new request from a bot. There will never be a logout from the bot to terminate the session, so each bot-related session will last for the time set for automatic expiry. Clearly it is inadvisable to bar bots, since most sites are anxious to gain search engine exposure. But it is possible to build session handling so as to limit the workload created by visitors who do not permit cookies, which will mostly be bots. When we move into implementation techniques, the mechanisms will be demonstrated. Session data and scalability We could simply let PHP take care of session data. It does that by writing a serialized version of any data placed into $_SESSION into a file in a temporary directory. Each session has its own file. But PHP also allows us to implement our own session data handling mechanism. There are a couple of good reasons for using that facility, and storing the information in the database. One is that we can analyze and manage the data better, and especially limit the overhead of dealing with search engine bots. The other is that by storing session data in the database, we make it feasible for the site to be run across multiple servers. There may well be other issues before that can be achieved, but providing session continuity is an essential requirement if load sharing is to be fully effective. Storing session data in a database is a reliable solution to this issue. Arguments against storing session data in a database include questions about the overhead involved, constraints on database performance, or the possibility of a single point of failure. While these are real issues, they can certainly be mitigated. Most database engines, including MySQL, have many options for building scalable and robust systems. If necessary, the database can be spread across multiple computers linked by a high speed network, although this should never be done unless it is really needed. Design of such a system is outside the scope of this article, but the key point is that the arguments against storing session data in a database are not particularly strong.

(For more resources related to this topic, see here.) The Hadoop platform Hadoop can be used for a lot of things. However, when you break it down to its core parts, the primary features of Hadoop are Hadoop Distributed File System (HDFS) and MapReduce. HDFS stores read-only files by splitting them into large blocks and distributing and replicating them across a Hadoop cluster. Two services are involved with the filesystem. The first service, the NameNode acts as a master and keeps the directory tree of all file blocks that exist in the filesystem and tracks where the file data is kept across the cluster. The actual data of the files is stored in multiple DataNode nodes, the second service. MapReduce is a programming model for processing large datasets with a parallel, distributed algorithm in a cluster. The most prominent trait of Hadoop is that it brings processing to the data; so, MapReduce executes tasks closest to the data as opposed to the data travelling to where the processing is performed. Two services are involved in a job execution. A job is submitted to the service JobTracker, which first discovers the location of the data. It then orchestrates the execution of the map and reduce tasks. The actual tasks are executed in multiple TaskTracker nodes. Hadoop handles infrastructure failures such as network issues, node, or disk failures automatically. Overall, it provides a framework for distributed storage within its distributed file system and execution of jobs. Moreover, it provides the service ZooKeeper to maintain configuration and distributed synchronization. Many projects surround Hadoop and complete the ecosystem of available Big Data processing tools such as utilities to import and export data, NoSQL databases, and event/real-time processing systems. The technologies that move Hadoop beyond batch processing focus on in-memory execution models. Overall multiple projects, from batch to hybrid and real-time execution exist. MapReduce Massive parallel processing of large datasets is a complex process. MapReduce simplifies this by providing a design pattern that instructs algorithms to be expressed in map and reduce phases. Map can be used to perform simple transformations on data, and reduce is used to group data together and perform aggregations. By chaining together a number of map and reduce phases, sophisticated algorithms can be achieved. The shared nothing architecture of MapReduce prohibits communication between map tasks of the same phase or reduces tasks of the same phase. Communication that's required happens at the end of each phase. The simplicity of this model allows Hadoop to translate each phase, depending on the amount of data that needs to be processed into tens or even hundreds of tasks being executed in parallel, thus achieving scalable performance. Internally, the map and reduce tasks follow a simplistic data representation. Everything is a key or a value. A map task receives key-value pairs and applies basic transformations emitting new key-value pairs. Data is then partitioned and different partitions are transmitted to different reduce tasks. A reduce task also receives key-value pairs, groups them based on the key, and applies basic transformation to those groups. A MapReduce example To illustrate how MapReduce works, let's look at an example of a log file of total size 1 GB with the following format: INFO MyApp - Entering application. WARNING com.foo.Bar - Timeout accessing DB - Retrying ERROR com.foo.Bar - Did it again! INFO MyApp - Exiting application Once this file is stored in HDFS, it is split into eight 128 MB blocks and distributed in multiple Hadoop nodes. In order to build a MapReduce job to count the amount of INFO, WARNING, and ERROR log lines in the file, we need to think in terms of map and reduce phases. In one map phase, we can read local blocks of the file and map each line to a key and a value. We can use the log level as the key and the number 1 as the value. After it is completed, data is partitioned based on the key and transmitted to the reduce tasks. MapReduce guarantees that the input to every reducer is sorted by key. Shuffle is the process of sorting and copying the output of the map tasks to the reducers to be used as input. By setting the value to 1 on the map phase, we can easily calculate the total in the reduce phase. Reducers receive input sorted by key, aggregate counters, and store results. In the following diagram, every green block represents an INFO message, every yellow block a WARNING message, and every red block an ERROR message: Implementing the preceding MapReduce algorithm in Java requires the following three classes: A Map class to map lines into <key,value> pairs; for example, <"INFO",1> A Reduce class to aggregate counters A Job configuration class to define input and output types for all <key,value> pairs and the input and output files MapReduce abstractions This simple MapReduce example requires more than 50 lines of Java code (mostly because of infrastructure and boilerplate code). In SQL, a similar implementation would just require the following: SELECT level, count(*) FROM table GROUP BY level Hive is a technology originating from Facebook that translates SQL commands, such as the preceding one, into sets of map and reduce phases. SQL offers convenient ubiquity, and it is known by almost everyone. However, SQL is declarative and expresses the logic of a computation without describing its control flow. So, there are use cases that will be unusual to implement in SQL, and some problems are too complex to be expressed in relational algebra. For example, SQL handles joins naturally, but it has no built-in mechanism for splitting data into streams and applying different operations to each substream. Pig is a technology originating from Yahoo that offers a relational data-flow language. It is procedural, supports splits, and provides useful operators for joining and grouping data. Code can be inserted anywhere in the data flow and is appealing because it is easy to read and learn. However, Pig is a purpose-built language; it excels at simple data flows, but it is inefficient for implementing non-trivial algorithms. In Pig, the same example can be implemented as follows: LogLine = load 'file.logs' as (level, message); LevelGroup = group LogLine by level; Result = foreach LevelGroup generate group, COUNT(LogLine); store Result into 'Results.txt'; Both Pig and Hive support extra functionality through loadable user-defined functions (UDF) implemented in Java classes. Cascading is implemented in Java and designed to be expressive and extensible. It is based on the design pattern of pipelines that many other technologies follow. The pipeline is inspired from the original chain of responsibility design pattern and allows ordered lists of actions to be executed. It provides a Java-based API for data-processing flows. Developers with functional programming backgrounds quickly introduced new domain specific languages that leverage its capabilities. Scalding, Cascalog, and PyCascading are popular implementations on top of Cascading, which are implemented in programming languages such as Scala, Clojure, and Python. Introducing Cascading Cascading is an abstraction that empowers us to write efficient MapReduce applications. The API provides a framework for developers who want to think in higher levels and follow Behavior Driven Development (BDD) and Test Driven Development (TDD) to provide more value and quality to the business. Cascading is a mature library that was released as an open source project in early 2008. It is a paradigm shift and introduces new notions that are easier to understand and work with. In Cascading, we define reusable pipes where operations on data are performed. Pipes connect with other pipes to create a pipeline. At each end of a pipeline, a tap is used. Two types of taps exist: source, where input data comes from and sink, where the data gets stored. In the preceding image, three pipes are connected to a pipeline, and two input sources and one output sink complete the flow. A complete pipeline is called a flow, and multiple flows bind together to form a cascade. In the following diagram, three flows form a cascade: The Cascading framework translates the pipes, flows, and cascades into sets of map and reduce phases. The flow and cascade planner ensure that no flow or cascade is executed until all its dependencies are satisfied. The preceding abstraction makes it easy to use a whiteboard to design and discuss data processing logic. We can now work on a productive higher level abstraction and build complex applications for ad targeting, logfile analysis, bioinformatics, machine learning, predictive analytics, web content mining, and for extract, transform and load (ETL) jobs. By abstracting from the complexity of key-value pairs and map and reduce phases of MapReduce, Cascading provides an API that so many other technologies are built on. What happens inside a pipe Inside a pipe, data flows in small containers called tuples. A tuple is like a fixed size ordered list of elements and is a base element in Cascading. Unlike an array or list, a tuple can hold objects with different types. Tuples stream within pipes. Each specific stream is associated with a schema. The schema evolves over time, as at one point in a pipe, a tuple of size one can receive an operation and transform into a tuple of size three. To illustrate this concept, we will use a JSON transformation job. Each line is originally stored in tuples of size one with a schema: 'jsonLine. An operation transforms these tuples into new tuples of size three: 'time, 'user, and 'action. Finally, we extract the epoch, and then the pipe contains tuples of size four: 'epoch, 'time, 'user, and 'action. Pipe assemblies Transformation of tuple streams occurs by applying one of the five types of operations, also called pipe assemblies: Each: To apply a function or a filter to each tuple GroupBy: To create a group of tuples by defining which element to use and to merge pipes that contain tuples with similar schemas Every: To perform aggregations (count, sum) and buffer operations to every group of tuples CoGroup: To apply SQL type joins, for example, Inner, Outer, Left, or Right joins SubAssembly: To chain multiple pipe assemblies into a pipe To implement the pipe for the logfile example with the INFO, WARNING, and ERROR levels, three assemblies are required: The Each assembly generates a tuple with two elements (level/message), the GroupBy assembly is used in the level, and then the Every assembly is applied to perform the count aggregation. We also need a source tap to read from a file and a sink tap to store the results in another file. Implementing this in Cascading requires 20 lines of code; in Scala/Scalding, the boilerplate is reduced to just the following: TextLine(inputFile) .mapTo('line->'level,'message) { line:String => tokenize(line) } .groupBy('level) { _.size } .write(Tsv(outputFile)) Cascading is the framework that provides the notions and abstractions of tuple streams and pipe assemblies. Scalding is a domain-specific language (DSL) that specializes in the particular domain of pipeline execution and further minimizes the amount of code that needs to be typed. Cascading extensions Cascading offers multiple extensions that can be used as taps to either read from or write data to, such as SQL, NoSQL, and several other distributed technologies that fit nicely with the MapReduce paradigm. A data processing application, for example, can use taps to collect data from a SQL database and some more from the Hadoop file system. Then, process the data, use a NoSQL database, and complete a machine learning stage. Finally, it can store some resulting data into another SQL database and update a mem-cache application. Summary This article explains the core technologies used in the distributed model of Hadoop Resources for Article: Further resources on this subject: Analytics – Drawing a Frequency Distribution with MapReduce (Intermediate) [article] Understanding MapReduce [article] Advanced Hadoop MapReduce Administration [article]

(For more resources related to this topic, see here.) Step 1 – fresh start In a new folder called, for example, 1_PubSub_Chat, let's open our editor of choice and create here a file called pubsub_chat.js. Also, make sure that you have a terminal window open and you moved into the newly created project directory. Step 2 – creating the TCP server TCP servers are called net servers in Vert.x. Creating and using a net server is really similar to HTTP servers: Obtain the vertx bridge object to access the framework features: var vertx = require('vertx'); /* 1 */var netServer = vertx.createNetServer(); /* 2 */netServer.listen(1234); /* 3 */ Ask Vert.x to create a TCP server (called NetServer in Vert.x). Actually, start the server by telling it to listen on TCP port 1234. Let's test whether this works. This time we need another terminal to run the telnet command: $ telnet localhost 1234 The terminal should now be connected and waiting to send/receive characters. If you have "connection refused" errors, make sure the server is running. Step 3 – adding a connect handler Now, we need to place a block of code to be executed as soon as a client connects: Define a handler function. This function will be called every time a client connects to the server: var vertx = require('vertx')var server = vertx.createNetServer().connectHandler(function(socket) {// Composing a client address stringaddr = socket.remoteAddress();addr = addr.ipaddress + addr.port;socket.write('Welcome to the chat ' + addr + '!');}).listen(1234) A NetServer connect handler accepts the socket object as a parameter; this object is our gateway to reading, writing, or closing the connection to a client. Use the socket object to write a greeting to the newly connected clients. If we test this one as in step 2 (Step 2 – creating the TCP server), we see that the server now welcomes us with a message containing an identifier of the client as its origin host and origin port. Step 4 – adding a data handler We just learned how to execute a block of code at the moment in which the client connects. However now we are interested in doing something else at the time when we receive new data from a client connection. The socket object we used in the previous step for writing data back to the client, accepts a handler function too: the data handler. Let's add one: Add a data handler function to the socket object. This is going to be called every time the client sends a new string of data: var vertx = require('vertx') var server = vertx.createNetServer().connectHandler( function(socket) { // Composing a client address string addr = socket.remoteAddress(); addr = addr.ipaddress + addr.port; socket.write('Welcome to the chat ' + addr + '!'); socket.dataHandler(function(data) { var now = new Date(); now = now.getHours() + ':' + now.getMinutes() + ':' + now.getSeconds(); var msg = now + ' <' + addr + '> ' + data; socket.write(msg); }) }).listen(1234) React to the new data event by writing the same data back to the socket (plus a prefix). What we have now is a sort of an echo server, which returns back to the sender the same message with a prefix string. Step 5 – adding the event bus magic The base requirement of a chat server is that every time a client sends a message, the rest of the connected clients should receive it. We will use event bus, the messaging service provided by the framework, to send (publish) received messages to a broadcast address. Each client will subscribe to the address upon connection and receive other clients' messages from there: Add a data handler function to the socket object: var vertx = require('vertx') var server = vertx.createNetServer().connectHandler( function(socket) { // Composing a client address string addr = socket.remoteAddress(); addr = addr.ipaddress + addr.port; socket.write('Welcome to the chat ' + addr + '!'); vertx.eventBus.registerHandler('broadcast_address', function(event){ socket.write(event); }); socket.dataHandler(function(data) { var now = new Date(); now = now.getHours() + ':' + now.getMinutes() + ':' + now.getSeconds(); var msg = now + ' <' + addr + '> ' + data; vertx.eventBus.publish('broadcast_address', msg); }) }).listen(1234) As soon as a client connects, they listen to the event bus for new data to be published on the address broadcast_address. When a client sends a string of characters to the server, this data is published to the broadcast address, triggering a handler function that writes the string to all the clients' sockets. The chat server is now complete! To try it out, just open three terminals: Terminal 1: $ vertx run pubsub_chat.js Terminal 2: $ telnet localhost 1234 Terminal 3: $ telnet localhost 1234 Now, we have a server and two clients running and connected. Type something in terminal 2 or 3 and see the message being broadcast to both the other windows: $ telnet localhost 1234Trying ::1...Connected to localhost.Escape character is '^]'.Hello from terminal two!13:6:56 <0:0:0:0:0:0:0:155991> Hello from terminal two!13:7:24 <0:0:0:0:0:0:0:155992> Hi there, here's terminal three!13:7:56 <0:0:0:0:0:0:0:155992> Great weather today! Step 6 – Organizing a more complex project Since Vert.x is a polyglot platform, we can choose to write an application (or a part of it) in either of the many supported languages. The granularity of the language choice is at verticle level. It's important to give a good architecture to a non-trivial project from the beginning. Follow this list of generic principles to avoid performance bottlenecks or the need for massive refactoring in the future: Wrap synchronous libraries or legacy code inside a worker verticle (or a module). This will keep the blocking code away from the event loop threads. Divide the problem in isolated domains and write a verticle to handle each of them (for example, database persistor verticle, web server verticle, authenticator verticle, and cache manager verticle). Use a startup verticle. This will be the single entry point to the application. Its responsibilities will be to: Validate the configuration file Programmatically deploy other verticles in the correct order Decide how many instances of a verticle to create (the decision might depend on the environment: for example, the amount of available processors) Register periodic tasks Summary: In this article, we learned in a step-wise procedure how we can create an Internet Relay Chat using the TCP server, and interconnect the server with the clients using an event bus, and enable different types of communication between them. Resources for Article: Further resources on this subject: Getting Started with Zombie.js [Article] Building a Chat Application [Article] Accessing and using the RDF data in Stanbol [Article]