Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-adding-and-editing-content-your-web-pages
Packt
06 Apr 2015
12 min read
Save for later

Adding and Editing Content in Your Web Pages

Packt
06 Apr 2015
12 min read
This article by Miko Coffey, the author of the book, Building Business Websites with Squarespace 7, delves into the processes of adjusting images, adding content to sidebars or footers, and adding links. (For more resources related to this topic, see here.) Adjusting images in Squarespace We've learned how to adjust the size of images in relation to other elements on the page, but so far, the images themselves have remained intact, showing the full image. However, you can actually crop or zoom images so they only show a portion of the photo on the screen without having to leave the Squarespace interface. You can also apply effects to images using the built-in Aviary Image Editor, such as rotating, enhancing color, boosting contrast, whitening teeth, removing blemishes, or hundreds of other adjustments, which means you don't need fancy image editing software to perform even fairly advanced image adjustments. Cropping and zooming images with LayoutEngine If you only want to crop your image, you don't need to use the Aviary Image Editor: you can crop images using LayoutEngine in the Squarespace Content Editor. To crop an image, you perform the same steps as those to adjust the height of a Spacer Block: just click and drag the dot to change the part of the image that is shown. As you drag the dot up or down, you will notice that: Dragging the dot up will chop off the top and bottom of your image Dragging the dot down will zoom in your image, cutting off the sides and making the image appear larger When dragging the dot very near the original dimensions of your image, you will feel and see the cursor pull/snap to the original size Cropping an image in an Image Block in this manner does not remove parts from the original image; it merely adjusts the part of the image that will be shown in the Image Block on the page. You can always change your mind later. Adjusting the Focal Point of images You'll notice that all of the cropping and zooming of images is based on the center of the image. What if your image has elements near the edges that you want to show instead of weighting things towards the center? With Squarespace, you can influence which part of the image displays by adjusting the Focal Point of the image. The Focal Point identifies the most important part of the image to instruct the system to try to use this point as the basis for cropping or zooming. However, if your Image Block is an extreme shape, such as a long skinny rectangle, it may not be possible to fit all of your desired area into the cropped or zoomed image space. Adjusting the Focal Point can also be useful for Gallery images, as certain templates display images in a square format or other formats that may not match the dimensions of your images. You can also adjust the Focal Point of any Thumbnail Images that you have added in Page Settings to select which part to show as the thumbnail or header banner. To adjust an image's Focal Point, follow these steps: Double-click on the image to open the Edit Image overlay window. Hover your mouse over the image thumbnail, and you will see a translucent circle appear at the center of the thumbnail. This is the Focal Point. Click and drag the circle until it sits on top of the part of the image you want to include, as shown in the following screenshot: Using the Aviary Image Editor You can also use the Aviary Image Editor to crop or zoom into your images as well as many more adjustments that are too numerous to list here. It's important to remember that all adjustments carried out in the Aviary Image Editor are permanent: there is no way to go back to a previous version of your image. Therefore, it's better to use LayoutEngine for cropping and zooming and reserve Aviary for other adjustments that you know you want to make permanently, such as rotating a portrait image that was taken sideways to display vertically. Because edits performed in the Aviary Image Editor are permanent, use it with caution and always keep a backup original version of the image on your computer just in case. Here's how to edit an image with the Aviary Image Editor: Double-click on the image to open the Edit Image overlay window. Click on the Edit button below the image thumbnail. This will open the Aviary window, as shown in the following screenshot: Select the type of adjustment you want to perform from the menu at the top. Use the controls to perform the adjustment and click on Apply. The window will show you the effect of the adjustment on the image. Perform any other adjustments in the same manner. You can go back to previous steps using the arrows in the bottom-left section of the editor window. Once you have performed all desired adjustments, click on Save to commit the adjustments to the image permanently. The Aviary window will now close. In the Edit Image window, click on Save to store the Aviary adjustments. The Edit Image window will now close. In the Content Editor window, click on the Save button to refresh the page with the newly edited version of the image. Adding content to sidebars or footers Until this point, all of our content additions and edits have been performed on a single page. However, it's likely that you will want to have certain blocks of content appear on multiple or all pages, such as a copyright notice in your page footer or an About the Author text snippet in the sidebar of all blog posts. You add content to footers or sidebars using the Content Editor in much the same way as adding the page content. However, there are a few restrictions. Certain templates only allow certain types of blocks in footers or sidebars, and some templates have restrictions on positioning elements as well—for example, it's unlikely that you will be able to wrap text around an image in a sidebar due to space limitations. If you are unable to get the system to accept an addition or repositioning move that you are trying to perform to a block in a sidebar or footer, it usually indicates that you are trying to perform something that is prohibited. Adding or editing content in a footer Follow these steps to add or edit content in a footer: In the preview screen, scroll to the bottom of your page and hover over the footer area to activate the Annotations there, and click on the Edit button next to the Footer Content label, as shown in the following screenshot: This will open the Content Editor window. You will notice that Insert Points appear just like before, and you can click on an Insert Point to add a block, or click within an existing Text Block to edit it. You can move blocks in a footer in the same way as those in a page body. Most templates have a footer, but not all of them do. Some templates hide the footer on certain page types, so if you can't see your footer, try looking at a standard page, or double-check whether your template offers one. Adding or editing content in a sidebar Not all templates have a sidebar, but if yours does, here's how you can add or edit content in your sidebar: While in the Pages menu, navigate to a page that you know has a sidebar in your template, such as a Blog page. You should see the template's demo content preloaded into your sidebar. Hover your mouse over the sidebar area to activate the Annotations, and click on the Edit button that appears at the top of the sidebar area. Make sure you click on the correct Annotation. Other Annotations may be activated on the page, so don't get confused and click on anything other than the sidebar Annotations, as shown in the following screenshot: Once the Content Editor window opens, you can click on an Insert Point to add a block or click within an existing Text Block to edit it. You can move blocks in a sidebar in the same way as those in a page body. Enabling a sidebar If you do not see the sidebar, but you know that your template allows one, and you know you are on the correct page type (for example, a Blog post), then it's possible that your sidebar is not enabled. Depending on the template, you enable your sidebar in one of two ways. The first method is in the Style Editor, as follows: First, ensure you are looking at a Blog page (or any page that can have a sidebar in your template). From the Home menu in the side panel, navigate to Design | Style Editor. In the Style Editor menu, scroll down until you see a set of controls related to Blog Styles. Find the control for the sidebar, and select the position you want. The following screenshot shows you an example of this: Click on Save to commit your changes. If you don't see the sidebar control in the Style Editor, you may find it in the Blog or Page Settings instead, as described here: First, ensure you are looking at the Blog page (or any page that can have a sidebar in your template), and then click on the Settings button in the Annotations or the cog icon in the Pages menu to open the Settings window. Look for a menu item called Page Layout and select the sidebar position, as shown in the following screenshot: On smaller screens, many templates use fluid layout to stack the sidebar below the main content area instead of showing it on the left- or right-hand side. If you can't see your sidebar and are viewing the website on a tablet or another small/low-resolution screen, scroll down to the bottom of the page and you will most likely see your sidebar content there, just above the footer. Adding links that point to web pages or files The final type of basic content that we'll cover in this article is adding hyperlinks to your pages. You can use these links to point to external websites, other pages on your own website, or files that visitors can either view within the browser or download to their computers. You can assign a link to any word or phrase in a Text Block, or you can assign a link to an image. You can also use a special type of block called a Button to make links really stand out and encourage users to click. When creating links in any of these scenarios, you will be presented with three main options: External: You can paste or type the full web address of the external website you want the link to point to. You can also choose to have this website open in a new window to allow users to keep your site open instead of navigating away entirely. The following screenshot shows the External option: Files: You can either upload a file directly, or link to a file that you have already uploaded earlier. This screenshot shows the Files option: Content: You can link to any page, category, or tag that you have created on your site. Linking to a category or tag will display a list of all items that have been labeled with that tag or category. Here's an example of the Content option: Assigning a link to word(s) in a Text Block Follow these steps to assign a link to a word or phrase in a Text Block: Highlight the word(s), and then click on the link icon (two interlocked ovals) in the text editor menu. Select the type of link you want to add, input the necessary settings, and then click anywhere outside the Edit Link window. This will close the Edit Link window, and you will see that the word is now a different color to indicate that the link has been applied. Click on the Save button at the top of the page. You can change or remove the link at any time by clicking on the word. A floating window will appear to show you what the link currently points to, along with the options to edit or remove the link. This is shown in the following screenshot: Assigning a link to an image Here's how you can assign a link to an image: Double-click on the image to open the Edit Image window. Under Clickthrough URL, select the type of link that you want to add and input the necessary settings. Click on Save. Creating a Button on your page You can create a Button on your page by following these steps: In the Content Editor window, find the point where you want to insert the button on the page, and click on the Insert Point to open the Add Block menu. Under the Filters & Lists category, choose Button. Type the text that you want to show on the button, select the type of link, and select the button size and alignment you want. The following screenshot shows the Edit Button window: This is how the button appears on the page: Summary In this article, you have acquired skills you need to add and edit basic web content, and you have learned how to move things around to create finished web pages with your desired page layout. Visit www.square-help.com/inspiration if you'd like to see some examples of different page layouts to give you ideas for making your own pages. Here, you'll see how you can use LayoutEngine to create sophisticated layouts for a range of page types. Resources for Article: Further resources on this subject: Welcoming your Visitors: Creating Attractive Home Pages and Overview Pages [article] Selecting Elements [article] Creating Blog Content in WordPress [article]
Read more
  • 0
  • 0
  • 11464

article-image-security-microsoft-azure
Packt
06 Apr 2015
9 min read
Save for later

Security in Microsoft Azure

Packt
06 Apr 2015
9 min read
In this article, we highlight some security points of interest, according to the ones explained in the book Microsoft Azure Security, by Roberto Freato. Microsoft Azure is a comprehensive set of services, which enable Cloud computing solutions for enterprises and small businesses. It supports a variety of tools and languages, providing users with building blocks that can be composed as needed. Azure is actually one of the biggest players in the Cloud computing market, solving scalability issues, speeding up the entire management process, and integrating with the existing development tool ecosystem. (For more resources related to this topic, see here.) Standards and Azure It is probably well known that the most widely accepted principles of IT security are confidentiality, integrity, and availability. Despite many security experts defining even more indicators/principles related to IT security, most security controls are focused on these principles, since the vulnerabilities are often expressed as a breach of one (or many) of these three. These three principles are also known as the CIA triangle: Confidentiality: It is about disclosure. A breach of confidentiality means that somewhere, some critical and confidential information has been disclosed unexpectedly. Integrity: It is about the state of information. A breach of integrity means that information has been corrupted or, alternatively, the meaning of the information has been altered unexpectedly. Availability: It is about interruption. A breach of availability means that information access is denied unexpectedly. Ensuring confidentiality, integrity, and availability means that information flows are always monitored and the necessary controls are enforced. To conclude, this is the purpose of a Security Management System, which, when talking about IT, becomes Information Security Management System (ISMS). The International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC) often work together to build international standards for specific technical fields. They released the ISO/IEC 27000 series to provide a family of standards for ISMS, starting from definitions (ISO/IEC 27000), up to governance (ISO/IEC 27014), and even more. Two standards of particular interests are the ISO/IEC 27001 and the ISO/IEC 27002. Microsoft manages the Azure infrastructure, At the most, users can manage the operating system inside a Virtual Machine (VM), but they do not need to administer, edit, or influence the under-the-hood infrastructure. They should not be able to do this at all. Therefore, Azure is a shared environment. This means that a customer's VM can run on the physical server of another customer and, for any given Azure service, two customers can even share the same VM (in some Platform as a Service (PaaS) and Software as a Service (SaaS) scenarios). The Microsoft Azure Trust Center (http://azure.microsoft.com/en-us/support/trust-center/) highlights the attention given to the Cloud infrastructure, in terms of what Microsoft does to enforce security, privacy, and compliance. Identity and Access Management It is very common that different people within the same organization would access and use the same Azure resources. In this case, a few scenarios arise: with the current portal, we can add several co-administrators; with the Preview portal, we can define fine-grained ACLs with the Role-Based Access Control (RBAC) features it implements. By default, we can add external users into the Azure Active Directory (AD), by inviting them through their e-mail address, which must be either a Microsoft account or an Azure AD account. In the Preview portal, the following hierarchy is, as follows: Subscription: It is the permission given at the subscription level, which is valid for each object within the subscription (that is, a Reader subscription can view everything within the subscription). Resource group: This is a fairly new concept of Azure. A resource group is (as the name suggests) a group or resources logically connected, as the collection of resources used for the same web project (a web hosting plan, an SQL server, and so on). Permission given at this level is valid for each object within the resource group. Individual resource: It is the permission given to an individual resource, and is valid only for that resource (that is, giving read-only access to a client to view the Application Insights of a website). Despite it resembles from its name, Azure AD is just an Identity and Access Management (IAM) service, managed and hosted by Microsoft in Azure. We should not even try to make a comparison, because they have different scopes and features. It is true that we can link Azure AD with an on-premise AD, but only for the purpose of extending its functionalities to work with Internet-based applications. Azure AD can be considered a SaaS for IAM before its relationship with Azure Services. A company that is offers its SaaS solution to clients, can also use Azure AD as the Identity Provider, relying on the several existing users of Office 365 (which relies on Azure AD for authentication) or Azure AD itself. Access Control Service (ACS) has been famous for a while for its capability to act as an identity bridge between applications and social identities. In the last few years, if developers wanted to integrate Facebook, Google, Yahoo, and Microsoft accounts (Live ID), they would have probably used ACS. Using Platform as a Service Although there are several ways to host custom code on Azure, the two most important building blocks are Websites and Cloud services. The first is actually a PaaS built on top of the second (a PaaS too), and uses an open source engine named Project Kudu (https://github.com/projectkudu/kudu). Kudu is an open source engine, which works with IIS and manages automatic or manual deployments of Azure Websites in a sandboxed environment. Kudu can also run outside Azure, but it is primarily supported to enable Website services. An Azure Cloud service is a container of roles: a role is the representation of a unit of execution and it can be a worker role (an arbitrary application) or a web role (an IIS application). Each role within a Cloud service can be deployed to several VMs (instances) at the same time, to provide scalability and load-balancing. From the security perspective, we need to pay attention to these aspects: Remote endpoints Remote desktops Startup tasks Microsoft Antimalware Network communication Azure Websites are some of the most advanced PaaS in the Cloud computing market, providing users with a lock-in free solution to run applications built in various languages/platforms. From the security perspective, we need to pay attention to these aspects: Credentials Connection modes Settings and connection strings Backups Extensions Azure services have grown much faster (with regard to the number of services and the surface area) than in the past, at an amazingly increasing rate: consequently, we have several options to store any kind of data (relational, NoSQL, binary, JSON, and so on). Azure Storage is the base service for almost everything on the platform. Storage security is implemented in two different ways: Account Keys Shared Access Signatures While looking at the specifications of many Azure Services, we often see the scalability targets section. For a given service, Azure provides users with a set of upper limits, in terms of capacity, bandwidth, and throughput to let them design their Cloud solutions better. Working with SQL Database is straightforward. However, a few security best practices must be implemented to improve security: Setting up firewall rules Setting up users and roles Connection settings Modern software architectures often rely on an in-memory caching system to save frequently accessed data that do not change too often. Some extreme scenarios require us to use an in-memory cache as the primary data store for sensitive data, pursuing design patterns oriented to eventual persistence and consistency. Azure Managed Cache is the evolution of the former AppFabric Cache for Windows servers and it is a managed by an in-memory cache service. Redis is an open source, high performance data store written in ANSI C: since its name stands for Remote Dictionary Server, it is a key value data store with optional durability. Azure Key Vault is a new and promising service that is used to store cryptographic keys and application secrets. There is an official library to operate against Key Vault from .NET, using Azure AD authentication to get secrets or use the keys. Before using it, it is necessary to set appropriate permissions on the Key Vault for external access, using the Set-AzureKeyVaultAccessPolicy command. Using Infrastructure as a Service Customers choosing Infrastructure as a Service (IaaS) usually have existing project constraints, which are not adaptive to PaaS. We can think about a complex installation of an enterprise-level software suite, such as ERP or a SharePoint farm. This is one of the cases where a service, such as an Azure Website, probably cannot fit. There two main services where the security requirements should be correctly understood and addressed are: Azure Virtual Machines Azure Virtual Networks VMs are the most configurable execution environments for applications that Azure provides. With VMs, we can run arbitrary workloads and run custom tools and applications, but we need to manage and maintain them directly, including the security. From the security perspective, we need to pay attention to these aspects: VM creation Endpoints and ACLs Networking and isolation Microsoft Antimalware Operating system firewalls Auditing and best practices Azure Backup helps protect servers or clients against data loss, providing a second place backup solution. While performing a backup, in fact, one of the primary requirements is the location of the backup: avoid backing up sensitive or critical data to a physical location that is strictly connected to the primary source of the data itself. In case of a disaster, if you involve the facility where the source is located, a higher probability of losing data (including the backup) can occur. Summary In this article we covered the various security related aspects of Microsoft Azure with services, such as PaaS, IaaS, and IAM. Resources for Article: Further resources on this subject: Web API and Client Integration [article] Setting up of Software Infrastructure on the Cloud [article] Windows Phone 8 Applications [article]
Read more
  • 0
  • 0
  • 11699

article-image-getting-started-odoo-development
Packt
06 Apr 2015
14 min read
Save for later

Getting Started with Odoo Development

Packt
06 Apr 2015
14 min read
In this article by Daniel Reis, author of the book Odoo Development Essentials, we will see how to get started with Odoo. Odoo is a powerful open source platform for business applications. A suite of closely integrated applications was built on it, covering all business areas from CRM and Sales to Accounting and Stocks. Odoo has a dynamic and growing community around it, constantly adding features, connectors, and additional business apps. Many can be found at Odoo.com. In this article, we will guide you to install Odoo from the source code and create your first Odoo application. Inspired by the todomvc.com project, we will build a simple to-do application. It should allow us to add new tasks, mark them as completed, and finally, clear the task list from all already completed tasks. (For more resources related to this topic, see here.) Installing Odoo from source We will use a Debian/Ubuntu system for our Odoo server, so you will need to have it installed and available to work on. If you don't have one, you might want to set up a virtual machine with a recent version of Ubuntu Server before proceeding. For a development environment, we will install it directly from Odoo's Git repository. This will end up giving more control on versions and updates. We will need to make sure Git is installed. In the terminal, type the following command: $ sudo apt-get update && sudo apt-get upgrade # Update system $ sudo apt-get install git # Install Git To keep things tidy, we will keep all our work in the /odoo-dev directory inside our home directory: $ mkdir ~/odoo-dev # Create a directory to work in $ cd ~/odoo-dev # Go into our work directory Now, we can use this script to show how to install Odoo from source code in a Debian system: $ git clone https://github.com/odoo/odoo.git -b 8.0 --depth=1 $ ./odoo/odoo.py setup_deps # Installs Odoo system dependencies $ ./odoo/odoo.py setup_pg   # Installs PostgreSQL & db superuser Quick start an Odoo instance In Odoo 8.0, we can create a directory and quick start a server instance for it. We can start by creating the directory called todo-app for our instance as shown here: $ mkdir ~/odoo-dev/todo-app $ cd ~/odoo-dev/todo-app Now we can create our todo_minimal module in it and initialize the Odoo instance: $ ~/odoo-dev/odoo/odoo.py scaffold todo_minimal $ ~/odoo-dev/odoo/odoo.py start -i todo_minimal The scaffold command creates a module directory using a predefined template. The start command creates a database with the current directory name and automatically adds it to the addons path so that its modules are available to be installed. Additionally, we used the -i option to also install our todo_minimal module. It will take a moment to initialize the database, and eventually we will see an INFO log message Modules loaded. Then, the server will be ready to listen to client requests. By default, the database is initialized with demonstration data, which is useful for development databases. Open http://<server-name>:8069 in your browser to be presented with the login screen. The default administrator account is admin with the password admin. Whenever you want to stop the Odoo server instance and return to the command line, press CTRL + C. If you are hosting Odoo in a virtual machine, you might need to do some network configuration to be able to use it as a server. The simplest solution is to change the VM network type from NAT to Bridged. Hopefully, this can help you find the appropriate solution in your virtualization software documentation. Creating the application models Now that we have an Odoo instance and a new module to work with, let's start by creating the data model. Models describe business objects, such as an opportunity, a sales order, or a partner (customer, supplier, and so on). A model has data fields and can also define specific business logic. Odoo models are implemented using a Python class derived from a template class. They translate directly to database objects, and Odoo automatically takes care of that when installing or upgrading the module. Let's edit the models.py file in the todo_minimal module directory so that it contains this: # -*- coding: utf-8 -*- from openerp import models, fields, api   class TodoTask(models.Model):    _name = 'todo.task'    name = fields.Char()    is_done = fields.Boolean()    active = fields.Boolean(default=True) Our to-do tasks will have a name title text, a done flag, and an active flag. The active field has a special meaning for Odoo; by default, records with a False value in it won't be visible to the user. We will use it to clear the tasks out of sight without actually deleting them. Upgrading a module For our changes to take effect, the module has to be upgraded. The simplest and fastest way to make all our changes to a module effective is to go to the terminal window where you have Odoo running, stop it (CTRL + C), and then restart it requesting the module upgrade. To start upgrading the server, the todo_minimal module in the todo-app database, use the following command: $ cd ~/odoo-dev/todo-app # we should be in the right directory $ ./odoo.py start -u todo_minimal The -u option performs an upgrade on a given list of modules. In this case, we upgrade just the todo_minimal module. Developing a module is an iterative process. You should make your changes in gradual steps and frequently install them with a module upgrade. Doing so will make it easier to detect mistakes sooner, and narrowing down the culprit in case the error message is not clear enough. And this can be very frequent when starting with Odoo development. Adding menu options Now that we have a model to store our data, let's make it available on the user interface. All we need is to add a menu option to open the to-do task model, so it can be used. This is done using an XML data file. Let's reuse the templates.xml data file and edit it so that it look like this: <openerp> <data>    <act_window id="todo_task_action"                name="To-do Task Action"                    res_model="todo.task" view_mode="tree,form" />        <menuitem id="todo_task_menu"                 name="To-do Tasks"                  action="todo_task_action"                  parent="mail.mail_feeds"                  sequence="20" /> </data> </openerp> Here, we have two records: a menu option and a window action. The Communication top menu to the user interface was added by the mail module dependency. We can know the identifier of the specific menu option where we want to add our own menu option by inspecting that module, it is mail_feeds. Also, our menu option executes the todo_task_action action we created. And that window action opens a tree view for the todo.task model. If we upgrade the module now and try the menu option just added, it will open an automatically generated view for our model, allowing to add and edit records. Views should be defined for models to be exposed to the users, but Odoo is nice enough to do that automatically if we don't, so we can work with our model right away, without having any form or list views defined yet. So far so good. Let's improve our user interface now. Creating Views Odoo supports several types of views, but the more important ones are list (also known as "tree"), form, and search views. For our simple module, we will just add a list view. Edit the templates.xml file to add the following <record> element just after the <data> opening tag at the top:    <record id="todo_task_tree" model="ir.ui.view">        <field name="name">To-do Task Form</field>        <field name="model">todo.task</field>        <field name="arch" type="xml">            <tree editable="top" colors="gray:is_done==True">                <field name="name" />                <field name="is_done" />            </tree>          </field>    </record> This creates a tree view for the todo.task model with two columns: the title name and the is_done flag. Additionally, it has a color rule to display the tasks done in gray. Adding business logic We want to add business logic to be able to clear the already completed tasks. Our plan is to add an option on the More button, shown at the top of the list when we select lines. We will use a very simple wizard for this, opening a confirmation dialog, where we can execute a method to inactivate the done tasks. Wizards use a special type of model for temporary data: a Transient model. We will now add it to the models.py file as follows: class TodoTaskClear(models.TransientModel):    _name = 'todo.task.clear'      @api.multi    def do_clear_done(self):        Task = self.env['todo.task']        done_recs = Task.search([('is_done', '=', True)])        done_recs.write({'active': False)}        return True Transient models work just like regular models, but their data is temporary and will eventually be purged from the database. In this case, we don't need any fields, since no additional input is going to be asked to the user. It just has a method that will be called when the confirmation button is pressed. It lists all tasks that are done and then sets their active flag to False. Next, we need to add the corresponding user interface. In the templates.xml file, add the following code:    <record id="todo_task_clear_dialog" model="ir.ui.view">      <field name="name">To-do Clear Wizard</field>      <field name="model">todo.task.clear</field>      <field name="arch" type="xml">        <form>           All done tasks will be cleared, even if            unselected.<br/>Continue?            <footer>                <button type="object"                        name="do_clear_done"                        string="Clear                        " class="oe_highlight" />                or <button special="cancel"                            string="Cancel"/>            </footer>        </form>      </field>    </record>      <!-- More button Action -->    <act_window id="todo_task_clear_action"        name="Clear Done"        src_model="todo.task"        res_model="todo.task.clear"        view_mode="form"        target="new" multi="True" /> The first record defines the form for the dialog window. It has a confirmation text and two buttons on the footer: Clear and Cancel. The Clear button when pressed will call the do_clear_done() method defined earlier. The second record is an action that adds the corresponding option in the More button for the to-do tasks model. Configuring security Finally, we need to set the default security configurations for our module. These configurations are usually stored inside the security/ directory. We need to add them to the __openerp__.py manifest file. Change the data attribute to the following: 'data': [  'security/ir.model.access.csv',    'security/todo_access_rules.xml',    'templates.xml'], The access control lists are defined for models and user groups in the ir.model.access.csv file. There is a pre-generated template. Edit it to look like this: id,name,model_id:id,group_id:id,perm_read,perm_write,perm_create,perm_unlink access_todo_task_user,To-do Task User Access,model_todo_task,base.group_user,1,1,1,1 This gives full access to all users in the base group named Employees. However, we want each user to see only their own to-do tasks. For that, we need a record rule setting a filter on the records the base group can see. Inside the security/ directory, add a todo_access_rules.xml file to define the record rule: <openerp> <data>    <record id="todo_task_user_rule" model="ir.rule">        <field name="name">ToDo Tasks only for owner</field>        <field name="model_id" ref="model_todo_task"/>        <field name="domain_force">[('create_uid','=',user.id)]              </field>        <field name="groups"                eval="[(4, ref('base.group_user'))]"/>    </record> </data> </openerp> This is all we need to set up the module security. Summary We created a new module from start, covering the most frequently used elements in a module: models, user interface views, business logic in model methods, and access security. In the process, we got familiar with the module development process, involving module upgrades and application server restarts to make the gradual changes effective in Odoo. Resources for Article: Further resources on this subject: Making Goods with Manufacturing Resource Planning [article] Machine Learning in IPython with scikit-learn [article] Administrating Solr [article]
Read more
  • 0
  • 0
  • 7710

article-image-high-availability-protection-and-recovery-using-microsoft-azure
Packt
02 Apr 2015
23 min read
Save for later

High Availability, Protection, and Recovery using Microsoft Azure

Packt
02 Apr 2015
23 min read
Microsoft Azure can be used to protect your on-premise assets such as virtual machines, applications, and data. In this article by Marcel van den Berg, the author of Managing Microsoft Hybrid Clouds, you will learn how to use Microsoft Azure to store backup data, replicate data, and even for orchestration of a failover and failback of a complete data center. We will focus on the following topics: High Availability in Microsoft Azure Introduction to geo-replication Disaster recovery using Azure Site Recovery (For more resources related to this topic, see here.) High availability in Microsoft Azure One of the most important limitations of Microsoft Azure is the lack of an SLA for single-instance virtual machines. If a virtual machine is not part of an availability set, that instance is not covered by any kind of SLA. The reason for this is that when Microsoft needs to perform maintenance on Azure hosts, in many cases, a reboot is required. Reboot means the virtual machines on that host will be unavailable for a while. So, in order to accomplish High Availability for your application, you should have at least two instances of the application running at any point in time. Microsoft is working on some sort of hot patching which enables virtual machines to remain active on hosts being patched. Details are not available at the moment of writing. High Availability is a crucial feature that must be an integral part of an architectural design, rather than something that can be "bolted on" to an application afterwards. Designing for High Availability involves leveraging both the development platform as well as available infrastructure in order to ensure an application's responsiveness and overall reliability. The Microsoft Azure Cloud platform offers software developers PaaS extensibility features and network administrators IaaS computing resources that enable availability to be built into an application's design from the beginning. The good news is that organizations with mission-critical applications can now leverage core features within the Microsoft Azure platform in order to deploy highly available, scalable, and fault-tolerant cloud services that have been shown to be more cost-effective than traditional approaches that leverage on-premises systems. Microsoft Failover Clustering support Windows Server Failover Clustering (WSFC) is not supported on Azure. However, Microsoft does support SQL Server AlwaysOn Availability Groups. For AlwaysOn Availability Groups, there is currently no support for availability group listeners in Azure. Also, you must work around a DHCP limitation in Azure when creating WSFC clusters in Azure. After you create a WSFC cluster using two Azure virtual machines, the cluster name cannot start because it cannot acquire a unique virtual IP address from the DHCP service. Instead, the IP address assigned to the cluster name is a duplicate address of one of the nodes. This has a cascading effect that ultimately causes the cluster quorum to fail, because the nodes cannot properly connect to one another. So if your application uses Failover Clustering, it is likely that you will not move it over to Azure. It might run, but Microsoft will not assist you when you encounter issues. Load balancing Besides clustering, we can also create highly available nodes using load balancing. Load balancing is useful for stateless servers. These are servers that are identical to each other and do not have a unique configuration or data. When two or more virtual machines deliver the same application logic, you will need a mechanism that is able to redirect network traffic to those virtual machines. The Windows Network Load Balancing (NLB) feature in Windows Server is not supported on Microsoft Azure. An Azure load balancer does exactly this. It analyzes incoming network traffic of Azure, determines the type of traffic, and reroutes it to a service.   The Azure load balancer is running provided as a cloud service. In fact, this cloud service is running on virtual appliances managed by Microsoft. These are completely software-defined. The moment an administrator adds an endpoint, a set of load balancers is instructed to pass incoming network traffic on a certain port to a port on a virtual machine. If a load balancer fails, another one will take over. Azure load balancing is performed at layer 4 of the OSI mode. This means the load balancer is not aware of the application content of the network packages. It just distributes packets based on network ports. To load balance over multiple virtual machines, you can create a load-balanced set by performing the following steps: In Azure Management Portal, select the virtual machine whose service should be load balanced. Select Endpoints in the upper menu. Click on Add. Select Add a stand-alone endpoint and click on the right arrow. Select a name or a protocol and set the public and private port. Enable create a load-balanced set and click on the right arrow. Next, fill in a name for the load-balanced set. Fill in the probe port, the probe interval, and the number of probes. This information is used by the load balancer to check whether the service is available. It will connect to the probe port number; do that according to the interval. If the specified number of probes all result in unable to connect, the load balancer will no longer distribute traffic to this virtual machine. Click on the check mark. The load balancing mechanism available is based on a hash. Microsoft Azure Load Balancer uses a five tuple (source IP, source port, destination IP, destination port, and protocol type) to calculate the hash that is used to map traffic to the available servers. A second load balancing mode was introduced in October 2014. It is called Source IP Affinity (also known as session affinity or client IP affinity). On using Source IP affinity, connections initiated from the same client computer go to the same DIP endpoint. These load balancers provide high availability inside a single data center. If a virtual machine part of a cluster of instances fails, the load balancer will notice this and remove that virtual machine IP address from a table. However, load balancers will not protect for failure of a complete data center. The domains that are used to direct clients to an application will route to a particular virtual IP that is bound to an Azure data center. To access application even if an Azure region has failed, you can use Azure Traffic Manager. This service can be used for several purposes: To failover to a different Azure region if a disaster occurs To provide the best user experience by directing network traffic to Azure region closest to the location of the user To reroute traffic to another Azure region whenever there's any planned maintenance The main task of Traffic Manager is to map a DNS query to an IP address that is the access point of a service. This job can be compared for example with a job of someone working with the X-ray machine at an airport. I'm guessing that you have all seen those multiple rows of X-ray machines. Each queue at an X-ray machine is different at any moment. An officer standing at the entry of the area distributes people over the available X-rays machine such that all queues remain equal in length. Traffic Manager provides you with a choice of load-balancing methods, including performance, failover, and round-robin. Performance load balancing measures the latency between the client and the cloud service endpoint. Traffic Manager is not aware of the actual load on virtual machines servicing applications. As Traffic Manager resolved endpoints of Azure cloud services only, it cannot be used for load balancing between an Azure region and a non-Azure region (for example, Amazon EC2) or between on-premises and Azure services. It will perform health checks on a regular basis. This is done by querying the endpoints of the services. If the endpoint does not respond, Traffic Manager will stop distributing network traffic to that endpoint for as long as the state of the endpoint is unavailable. Traffic Manager is available in all Azure regions. Microsoft charges for using this service based on the number of DNS queries that are received by Traffic Manager. As the service is attached to an Azure subscription, you will be required to contact Azure support to transfer Traffic Manager to a different subscription. The following table shows the difference between Azure's built-in load balancer and Traffic Manager:   Load balancer Traffic Manager Distribution targets Must reside in same region Can be across regions Load balancing 5 tuple Source IP Affinity  Performance, failover, and round-robin Level OSI layer 4 TCP/UDP ports OSI Layer 4 DNS queries Third-party load balancers In certain configurations, the default Azure load balancer might not be sufficient. There are several vendors supporting or starting to support Azure. One of them is Kemp Technologies. Kemp Technologies offers a free load balancer for Microsoft Azure. The Virtual LoadMaster (VLM) provides layer 7 application delivery. The virtual appliance has some limitations compared to the commercially available unit. The maximum bandwidth is limited to 100 Mbps and High Availability is not offered. This means the Kemp LoadMaster for Azure free edition is a single point of failure. Also, the number of SSL transactions per second is limited. One of the use cases in which a third-party load balancer is required is when we use Microsoft Remote Desktop Gateway. As you might know, Citrix has been supporting the use of Citrix XenApp and Citrix XenDesktop running on Azure since 2013. This means service providers can offer cloud-based desktops and applications using these Citrix solutions. To make this a working configuration, session affinity is required. Session affinity makes sure that network traffic is always routed over the same server. Windows Server 2012 Remote Desktop Gateway uses two HTTP channels, one for input and one for output, which must be routed over the same Remote Desktop Gateway. The Azure load balancer is only able to do round-robin load balancing, which does not guarantee both channels using the same server. However, hardware and software load balancers that support IP affinity, cookie-based affinity, or SSL ID-based affinity (and thus ensure that both HTTP connections are routed to the same server) can be used with Remote Desktop Gateway. Another use case is load balancing of Active Directory Federation Services (ADFS). Microsoft Azure can be used as a backup for on-premises Active Directory (AD). Suppose your organization is using Office 365. To provide single sign-on, a federation has been set up between Office 365 directory and your on-premises AD. If your on-premises ADFS fails, external users would not be able to authenticate. By using Microsoft Azure for ADFS, you can provide high availability for authentication. Kemp LoadMaster for Azure can be used to load balance network traffic to ADFS and is able to do proper load balancing. To install Kemp LoadMaster, perform the following steps: Download the Publish Profile settings file from https://windows.azure.com/download/publishprofile.aspx. Use PowerShell for Azure with the Import-AzurePublishSettingsFile command. Upload the KEMP supplied VHD file to your Microsoft Azure storage account. Publish the VHD as an image. The VHD will be available as an image. The image can be used to create virtual machines. The complete steps are described in the documentation provided by Kemp. Geo-replication of data Microsoft Azure has geo-replication of Azure Storage enabled by default. This means all of your data is not only stored at three different locations in the primary region, but also replicated and stored at three different locations at the paired region. However, this data cannot be accessed by the customer. Microsoft has to declare a data center or storage stamp as lost before Microsoft will failover to the secondary location. In the rare circumstance where a failed storage stamp cannot be recovered, you will experience many hours of downtime. So, you have to make sure you have your own disaster recovery procedures in place. Zone Redundant Storage Microsoft offers a third option you can use to store data. Zone Redundant Storage (ZRS) is a mix of two options for data redundancy and allows data to be replicated to a secondary data center / facility located in the same region or to a paired region. Instead of storing six copies of data like geo-replicated storage does, only three copies of data are stored. So, ZRS is a mix of local redundant storage and geo-replicated storage. The cost for ZRS is about 66 percent of the cost for GRS. Snapshots of the Microsoft Azure disk Server virtualization solutions such as Hyper-V and VMware vSphere offer the ability to save the state of a running virtual machine. This can be useful when you're making changes to the virtual machine but want to have the ability to reverse those changes if something goes wrong. This feature is called a snapshot. Basically, a virtual disk is saved by marking it as read only. All writes to the disk after a snapshot has been initiated are stored on a temporary virtual disk. When a snapshot is deleted, those changes are committed from the delta disk to the initial disk. While the Microsoft Azure Management Portal does not have a feature to create snapshots, there is an ability to make point-in-time copies of virtual disks attached to virtual machines. Microsoft Azure Storage has the ability of versioning. Under the hood, this works differently than snapshots in Hyper-V. It creates a snapshot blob of the base blob. Snapshots are by no ways a replacement for a backup, but it is nice to know you can save the state as well as quickly reverse if required. Introduction to geo-replication By default, Microsoft replicates all data stored on Microsoft Azure Storage to the secondary location located in the paired region. Customers are able to enable or disable the replication. When enabled, customers are charged. When Geo Redundant Storage has been enabled on a storage account, all data is asynchronous replicated. At the secondary location, data is stored on three different storage nodes. So even when two nodes fail, the data is still accessible. However, before the read access Geo-Redundant feature was available, customers had no way to actually access replicated data. The replicated data could only be used by Microsoft when the primary storage could not be recovered again. Microsoft will try everything to restore data in the primary location and avoid a so-called geo-failover process. A geo-failover process means that a storage account's secondary location (the replicated data) will be configured as the new primary location. The problem is that a geo-failover process cannot be done per storage account, but needs to be done at the storage stamp level. A storage stamp has multiple racks of storage nodes. You can imagine how much data and how many customers are involved when a storage stamp needs to failover. Failover will have an effect on the availability of applications. Also, because of the asynchronous replication, some data will be lost when a failover is performed. Microsoft is working on an API that allows customers to failover a storage account themselves. When geo-redundant replication is enabled, you will only benefit from it when Microsoft has a major issue. Geo-redundant storage is neither a replacement for a backup nor for a disaster recovery solution. Microsoft states that the Recover Point Objective (RPO) for Geo Redundant Storage will be about 15 minutes. That means if a failover is required, customers can lose about 15 minutes of data. Microsoft does not provide a SLA on how long geo-replication will take. Microsoft does not give an indication for the Recovery Time Objective (RTO). The RTO indicates the time required by Microsoft to make data available again after a major failure that requires a failover. Microsoft once had to deal with a failure of storage stamps. They did not do a failover but it took many hours to restore the storage service to a normal level. In 2013, Microsoft introduced a new feature called Read Access Geo Redundant Storage (RA-GRS). This feature allows customers to perform reads on the replicated data. This increases the read availability from 99.9 percent when GRS is used to above 99.99 percent when RA-GRS is enabled. Microsoft charges more when RA-GRS is enabled. RA-GRS is an interesting addition for applications that are primarily meant for read-only purposes. When the primary location is not available and Microsoft has not done a failover, writes are not possible. The availability of the Azure Virtual Machine service is not increased by enabling RA-GRS. While the VHD data is replicated and can be read, the virtual machine itself is not replicated. Perhaps this will be a feature for the future. Disaster recovery using Azure Site Recovery Disaster recovery has always been on the top priorities for organizations. IT has become a very important, if not mission-critical factor for doing business. A failure of IT could result in loss of money, customers, orders, and brand value. There are many situations that can disrupt IT such as: Hurricanes Floods Earthquakes Disasters such as a failure of a nuclear power plant Fire Human error Outbreak of a virus Hardware or software failure While these threads are clear and the risk of being hit by such a thread can be calculated, many organizations do not have a proper protection against those threads. In three different situations, disaster recovery solutions can help an organization to continue doing business: Avoiding a possible failure of IT infrastructure by moving servers to a different location. Avoiding a disaster situation, such as hurricanes or floods, since such situations are generally well known in advance due to weather forecasting capabilities. Recovering as quickly as possible when a disaster has hit the data center. Disaster recovery is done when a disaster unexpectedly hit the data center, such as a fire, hardware error, or human error. Some reasons for not having a proper disaster recovery plan are complexity, lack of time, and ignorance; however, in most cases, a lack of budget and the belief that disaster recovery is expensive are the main reasons. Almost all organizations that have been hit by a major disaster causing unacceptable periods of downtime started to implement a disaster recovery plan, including technology immediately after they recovered. However, in many cases, this insight came too late. According to Gartner, 43 percent of companies experiencing disasters never reopen and 29 percent close within 2 years. Server virtualization has made disaster recovery a lot easier and cost effective. Verifying that your DR procedure actually works as designed and matches RTO and RPO is much easier using virtual machines. Since Windows Server 2012, Hyper-V has a feature for asynchronous replication of virtual machine virtual disks to another location. This feature, Hyper-V Replica, is very easy to enable and configure. It does not cost extra. Hyper-V Replica is storage agnostic, which means the storage type at the primary site can be different than the storage type used in the secondary site. So, Hyper-V Replica perfectly works when your virtual machines are hosted on, for example, EMC storage while in the secondary a HP solution is used. While replication is a must for DR, another very useful feature in DR is automation. As an administrator, you really appreciate the option to click on a button after deciding to perform a failover and sit back and relax. Recovery is mostly a stressful job when your primary location is flooded or burned and lots of things can go wrong if recovery is done manually. This is why Microsoft designed Azure Site Recovery. Azure Site Recovery is able to assist in disaster recovery in several scenarios: A customer has two data centers both running Hyper-V managed by System Center Virtual Machine Manager. Hyper-V Replica is used to replicate data at the virtual machine level. A customer has two data centers both running Hyper-V managed by System Center Virtual Machine Manager. NetApp storage is used to replicate between two sites at the storage level. A customer has a single data center running Hyper-V managed by System Center Virtual Machine Manager. A customer has two data centers both running VMware vSphere. In this case InMage Scout software is used to replicate between two datacenters. Azure is not used for orchestration. A customer has a single data centers not managed by System Center Virtual Machine Manager. In the second scenario, Microsoft Azure is used as a secondary data center if a disaster makes the primary data center unavailable. Microsoft announced also to support a scenario where vSphere is used on-premises and Azure Site Recovery can be used to replicate data to Azure. To enable this InMage software will be used. Details were not available at the time this article was written. In the first two described scenarios Site Recovery is used to orchestrate the failover and failback to the secondary location. The management is done using Azure Management Portal. This is available using any browser supporting HTML5. So a failover can be initiated even from a tablet or smartphone. Using Azure as a secondary data center for disaster recovery Azure Site Recovery went into preview in June 2014. For organizations using Hyper-V, there is no direct need to have a secondary data center as Azure can be used as a target for Hyper-V Replica. Some of the characteristics of the service are as follows: Allows nondisruptive disaster recovery failover testing Automated reconfigure of network configuration of guests Storage agnostic supports any type of on-premises storage supported by Hyper-V Support for VSS to enable application consistency Protects more than 1,000 virtual machines (Microsoft tested with 2,000 virtual machines and this went well) To be able to use Site Recovery, customers do not have to use System Center Virtual Machine Manager. Site Recovery can be used without this installed. System Center Virtual Machine Manager. Site Recovery will use information such as? virtual networks provided by SCVMM to map networks available in Microsoft Azure. Site Recovery does not support the ability to send a copy of the virtual hard disks on removable media to an Azure data center to prevent the initial replication using WAN (seeding). Customers will need to transfer all the replication data over the network. ExpressRoute will help to get a much better throughput compared to a site-to-site VPN over the Internet. Failover to Azure can be as simple as clicking on a single button. Site Recovery will then create new virtual machines in Azure and start the virtual machines in the order defined in the recovery plan. A recovery plan is a workflow that defines the startup sequence of virtual machines. It is possible to stop the recovery plan to allow a manual check, for example. If all is okay, the recovery plan will continue doing its job. Multiple recovery plans can be created. Microsoft Volume Shadow Copy Services (VSS) is supported. This allows application consistency. Replication of data can be configured at intervals of 15 seconds, 5 minutes, or 15 minutes. Replication is performed asynchronously. For recovery, 24 recovery points are available. These are like snapshots or point-in-time copies. If the most recent replica cannot be used (for example, because of damaged data), another replica can be used for restore. You can configure extended replication. In extended replication, your Replica server forwards changes that occur on the primary virtual machines to a third server (the extended Replica server). After a planned or unplanned failover from the primary server to the Replica server, the extended Replica server provides further business continuity protection. As with ordinary replication, you configure extended replication by using Hyper-V Manager, Windows PowerShell (using the –Extended option), or WMI. At the moment, only VHD virtual disk format is supported. Generation 2 virtual machines that can be created on Hyper-V are not supported by Site Recovery. Generation 2 virtual machines have a simplified virtual hardware model and support Unified Extensible Firmware Interface (UEFI) firmware instead of BIOS-based firmware. Also, boot from PXE, SCSI hard disk, SCSCI DVD, and Secure Boot are supported in Generation 2 virtual machines. However on March 19 Microsoft responded to numerous customer requests on support of Site Recovery for Generation 2 virtual machines. Site Recovery will soon support Gen 2 VM's. On failover, the VM will be converted to a Gen 1 VM. On failback, the VM will be converted to Gen 2. This conversion is done till the Azure platform natively supports Gen 2 VM's. Customers using Site Recovery are charged only for consumption of storage as long as they do not perform a failover or failover test. Failback is also supported. After running for a while in Microsoft Azure customers are likely to move their virtual machines back to the on-premises, primary data center. Site Recovery will replicate back only the changed data. Mind that customer data is not stored in Microsoft Azure when Hyper-V Recovery Manager is used. Azure is used to coordinate the failover and recovery. To be able to do this, it stores information on network mappings, runbooks, and names of virtual machines and virtual networks. All data sent to Azure is encrypted. By using Azure Site Recovery, we can perform service orchestration in terms of replication, planned failover, unplanned failover, and test failover. The entire engine is powered by Azure Site Recovery Manager. Let's have a closer look on the main features of Azure Site Recovery. It enables three main scenarios: Test Failover or DR Drills: Enable support for application testing by creating test virtual machines and networks as specified by the user. Without impacting production workloads or their protection, HRM can quickly enable periodic workload testing. Planned Failovers (PFO): For compliance or in the event of a planned outage, customers can use planned failovers, virtual machines are shutdown, final changes are replicated to ensure zero data loss, and then virtual machines are brought up in order on the recovery site as specified by the RP. More importantly, failback is a single-click gesture that executes a planned failover in the reverse direction. Unplanned Failovers (UFO): In the event of unplanned outage or a natural disaster, HRM opportunistically attempts to shut down the primary machines if some of the virtual machines are still running when the disaster strikes. It then automates their recovery on the secondary site as specified by the RP. If your secondary site uses a different IP subnet, Site Recovery is able to change the IP configuration of your virtual machines during the failover. Part of the Site Recovery installation is the installation of a VMM provider. This component communicates with Microsoft Azure. Site Recovery can be used even if you have a single VMM to manage both primary and secondary sites. Site Recovery does not rely on availability of any component in the primary site when performing a failover. So it doesn't matter if the complete site including link to Azure has been destroyed, as Site Recovery will be able to perform the coordinated failover. Azure Site Recovery to customer owned sites is billed per protected virtual machine per month. The costs are approximately €12 per month. Microsoft bills for the average consumption of virtual machines per month. So if you are protecting 20 virtual machines in the first half and 0 in the second half, you will be charged for 10 virtual machines for that month. When Azure is used as a target, Microsoft will only charge for consumption of storage during replication. The costs for this scenario are €40.22/month per instance protected. As soon as you perform a test failover or actual failover Microsoft will charge for the virtual machine CPU and memory consumption. Summary Thus this article has covered the concepts of High Availability in Microsoft Azure and disaster recovery using Azure Site Recovery, and also gives an introduction to the concept of geo-replication. Resources for Article: Further resources on this subject: Windows Azure Mobile Services - Implementing Push Notifications using [article] Configuring organization network services [article] Integration with System Center Operations Manager 2012 SP1 [article]
Read more
  • 0
  • 0
  • 4023

article-image-bsp-layer
Packt
02 Apr 2015
14 min read
Save for later

The BSP Layer

Packt
02 Apr 2015
14 min read
In this article by Alex González, author of the book Embedded LinuxProjects Using Yocto Project Cookbook, we will see how the embedded Linux projects require both custom hardware and software. An early task in the development process is to test different hardware reference boards and the selection of one to base our design on. We have chosen the Wandboard, a Freescale i.MX6-based platform, as it is an affordable and open board, which makes it perfect for our needs. On an embedded project, it is usually a good idea to start working on the software as soon as possible, probably before the hardware prototypes are ready, so that it is possible to start working directly with the reference design. But at some point, the hardware prototypes will be ready and changes will need to be introduced into Yocto to support the new hardware. This article will explain how to create a BSP layer to contain those hardware-specific changes, as well as show how to work with the U-Boot bootloader and the Linux kernel, components which are likely to take most of the customization work. (For more resources related to this topic, see here.) Creating a custom BSP layer These custom changes are kept on a separate Yocto layer, called a Board Support Package (BSP) layer. This separation is best for future updates and patches to the system. A BSP layer can support any number of new machines and any new software feature that is linked to the hardware itself. How to do it... By convention, Yocto layer names start with meta, short for metadata. A BSP layer may then add a bsp keyword, and finally a unique name. We will call our layer meta-bsp-custom. There are several ways to create a new layer: Manually, once you know what is required By copying the meta-skeleton layer included in Poky By using the yocto-layer command-line tool You can have a look at the meta-skeleton layer in Poky and see that it includes the following elements: A layer.conf file, where the layer configuration variables are set A COPYING.MIT license file Several directories named with the recipes prefix with example recipes for BusyBox, the Linux kernel and an example module, an example service recipe, an example user management recipe, and a multilib example. How it works... We will cover some of the use cases that appear in the available examples, so for our needs, we will use the yocto-layer tool, which allows us to create a minimal layer. Open a new terminal and change to the fsl-community-bsp directory. Then set up the environment as follows: $ source setup-environment wandboard-quad Note that once the build directory has been created, the MACHINE variable has already been configured in the conf/local.conf file and can be omitted from the command line. Change to the sources directory and run: $ yocto-layer create bsp-custom Note that the yocto-layer tool will add the meta prefix to your layer, so you don't need to. It will prompt a few questions: The layer priority which is used to decide the layer precedence in cases where the same recipe (with the same name) exists in several layers simultaneously. It is also used to decide in what order bbappends are applied if several layers append the same recipe. Leave the default value of 6. This will be stored in the layer's conf/layer.conf file as BBFILE_PRIORITY. Whether to create example recipes and append files. Let's leave the default no for the time being. Our new layer has the following structure: meta-bsp-custom/    conf/layer.conf    COPYING.MIT    README There's more... The first thing to do is to add this new layer to your project's conf/bblayer.conf file. It is a good idea to add it to your template conf directory's bblayers.conf.sample file too, so that it is correctly appended when creating new projects. The highlighted line in the following code shows the addition of the layer to the conf/bblayers.conf file: LCONF_VERSION = "6"   BBPATH = "${TOPDIR}" BSPDIR := "${@os.path.abspath(os.path.dirname(d.getVar('FILE', "" True)) + '/../..')}"   BBFILES ?= "" BBLAYERS = " ${BSPDIR}/sources/poky/meta ${BSPDIR}/sources/poky/meta-yocto ${BSPDIR}/sources/meta-openembedded/meta-oe ${BSPDIR}/sources/meta-openembedded/meta-multimedia ${BSPDIR}/sources/meta-fsl-arm ${BSPDIR}/sources/meta-fsl-arm-extra ${BSPDIR}/sources/meta-fsl-demos ${BSPDIR}/sources/meta-bsp-custom " Now, BitBake will parse the bblayers.conf file and find the conf/layers.conf file from your layer. In it, we find the following line: BBFILES += "${LAYERDIR}/recipes-*/*/*.bb        ${LAYERDIR}/recipes-*/*/*.bbappend" It tells BitBake which directories to parse for recipes and append files. You need to make sure your directory and file hierarchy in this new layer matches the given pattern, or you will need to modify it. BitBake will also find the following: BBPATH .= ":${LAYERDIR}" The BBPATH variable is used to locate the bbclass files and the configuration and files included with the include and require directives. The search finishes with the first match, so it is best to keep filenames unique. Some other variables we might consider defining in our conf/layer.conf file are: LAYERDEPENDS_bsp-custom = "fsl-arm" LAYERVERSION_bsp-custom = "1" The LAYERDEPENDS literal is a space-separated list of other layers your layer depends on, and the LAYERVERSION literal specifies the version of your layer in case other layers want to add a dependency to a specific version. The COPYING.MIT file specifies the license for the metadata contained in the layer. The Yocto project is licensed under the MIT license, which is also compatible with the General Public License (GPL). This license applies only to the metadata, as every package included in your build will have its own license. The README file will need to be modified for your specific layer. It is usual to describe the layer and provide any other layer dependencies and usage instructions. Adding a new machine When customizing your BSP, it is usually a good idea to introduce a new machine for your hardware. These are kept under the conf/machine directory in your BSP layer. The usual thing to do is to base it on the reference design. For example, wandboard-quad has the following machine configuration file: include include/wandboard.inc   SOC_FAMILY = "mx6:mx6q:wandboard"   UBOOT_MACHINE = "wandboard_quad_config"   KERNEL_DEVICETREE = "imx6q-wandboard.dtb"   MACHINE_FEATURES += "bluetooth wifi"   MACHINE_EXTRA_RRECOMMENDS += " bcm4329-nvram-config bcm4330-nvram-config " A machine based on the Wandboard design could define its own machine configuration file, wandboard-quad-custom.conf, as follows: include conf/machine/include/wandboard.inc   SOC_FAMILY = "mx6:mx6q:wandboard"   UBOOT_MACHINE = "wandboard_quad_custom_config"   KERNEL_DEVICETREE = "imx6q-wandboard-custom.dtb"   MACHINE_FEATURES += "wifi" The wandboard.inc file now resides on a different layer, so in order for BitBake to find it, we need to specify the full path from the BBPATH variable in the corresponding layer. This machine defines its own U-Boot configuration file and Linux kernel device tree in addition to defining its own set of machine features. Adding a custom device tree to the Linux kernel To add this device tree file to the Linux kernel, we need to add the device tree file to the arch/arm/boot/dts directory under the Linux kernel source and also modify the Linux build system's arch/arm/boot/dts/Makefile file to build it as follows: dtb-$(CONFIG_ARCH_MXC) += +imx6q-wandboard-custom.dtb This code uses diff formatting, where the lines with a minus prefix are removed, the ones with a plus sign are added, and the ones without a prefix are left as reference. Once the patch is prepared, it can be added to the meta-bsp-custom/recipes-kernel/linux/linux-wandboard-3.10.17/ directory and the Linux kernel recipe appended adding a meta-bsp-custom/recipes-kernel/linux/linux-wandboard_3.10.17.bbappend file with the following content: SRC_URI_append = " file://0001-ARM-dts-Add-wandboard-custom-dts- "" file.patch" Adding a custom U-Boot machine In the same way, the U-Boot source may be patched to add a new custom machine. Bootloader modifications are not as likely to be needed as kernel modifications though, and most custom platforms will leave the bootloader unchanged. The patch would be added to the meta-bsp-custom/recipes-bsp/u-boot/u-boot-fslc-v2014.10/ directory and the U-Boot recipe appended with a meta-bsp-custom/recipes-bsp/u-boot/u-boot-fslc_2014.10.bbappend file with the following content: SRC_URI_append = " file://0001-boards-Add-wandboard-custom.patch" Adding a custom formfactor file Custom platforms can also define their own formfactor file with information that the build system cannot obtain from other sources, such as defining whether a touchscreen is available or defining the screen orientation. These are defined in the recipes-bsp/formfactor/ directory in our meta-bsp-custom layer. For our new machine, we could define a meta-bsp-custom/recipes-bsp/formfactor/formfactor_0.0.bbappend file to include a formfactor file as follows: FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:" And the machine-specific meta-bsp-custom/recipes-bsp/formfactor/formfactor/wandboard-quadcustom/machconfig file would be as follows: HAVE_TOUCHSCREEN=1 Debugging the Linux kernel booting process We have seen the most general techniques for debugging the Linux kernel. However, some special scenarios require the use of different methods. One of the most common scenarios in embedded Linux development is the debugging of the booting process. This section will explain some of the techniques used to debug the kernel's booting process. How to do it... A kernel crashing on boot usually provides no output whatsoever on the console. As daunting as that may seem, there are techniques we can use to extract debug information. Early crashes usually happen before the serial console has been initialized, so even if there were log messages, we would not see them. The first thing we will show is how to enable early log messages that do not need the serial driver. In case that is not enough, we will also show techniques to access the log buffer in memory. How it works... Debugging booting problems have two distinctive phases, before and after the serial console is initialized. After the serial is initialized and we can see serial output from the kernel, debugging can use the techniques described earlier. Before the serial is initialized, however, there is a basic UART support in ARM kernels that allows you to use the serial from early boot. This support is compiled in with the CONFIG_DEBUG_LL configuration variable. This adds supports for a debug-only series of assembly functions that allow you to output data to a UART. The low-level support is platform specific, and for the i.MX6, it can be found under arch/arm/include/debug/imx.S. The code allows for this low-level UART to be configured through the CONFIG_DEBUG_IMX_UART_PORT configuration variable. We can use this support directly by using the printascii function as follows: extern void printascii(const char *); printascii("Literal stringn"); However, much more preferred would be to use the early_print function, which makes use of the function explained previously and accepts formatted input in printf style; for example: early_print("%08xt%sn", p->nr, p->name); Dumping the kernel's printk buffer from the bootloader Another useful technique to debug Linux kernel crashes at boot is to analyze the kernel log after the crash. This is only possible if the RAM memory is persistent across reboots and does not get initialized by the bootloader. As U-Boot keeps the memory intact, we can use this method to peek at the kernel login memory in search of clues. Looking at the kernel source, we can see how the log ring buffer is set up in kernel/printk/printk.c and also note that it is stored in __log_buf. To find the location of the kernel buffer, we will use the System.map file created by the Linux build process, which maps symbols with virtual addresses using the following command: $grep __log_buf System.map 80f450c0 b __log_buf To convert the virtual address to physical address, we look at how __virt_to_phys() is defined for ARM: x - PAGE_OFFSET + PHYS_OFFSET The PAGE_OFFSET variable is defined in the kernel configuration as: config PAGE_OFFSET        hex        default 0x40000000 if VMSPLIT_1G        default 0x80000000 if VMSPLIT_2G        default 0xC0000000 Some of the ARM platforms, like the i.MX6, will dynamically patch the __virt_to_phys() translation at runtime, so PHYS_OFFSET will depend on where the kernel is loaded into memory. As this can vary, the calculation we just saw is platform specific. For the Wandboard, the physical address for 0x80f450c0 is 0x10f450c0. We can then force a reboot using a magic SysRq key, which needs to be enabled in the kernel configuration with CONFIG_MAGIC_SYSRQ, but is enabled in the Wandboard by default: $ echo b > /proc/sysrq-trigger We then dump that memory address from U-Boot as follows: > md.l 0x10f450c0 10f450c0: 00000000 00000000 00210038 c6000000   ........8.!..... 10f450d0: 746f6f42 20676e69 756e694c 6e6f2078   Booting Linux on 10f450e0: 79687020 61636973 5043206c 78302055     physical CPU 0x 10f450f0: 00000030 00000000 00000000 00000000   0............... 10f45100: 009600a8 a6000000 756e694c 65762078   ........Linux ve 10f45110: 6f697372 2e33206e 312e3031 2e312d37   rsion 3.10.17-1. 10f45120: 2d322e30 646e6177 72616f62 62672b64   0.2-wandboard+gb 10f45130: 36643865 62323738 20626535 656c6128   e8d6872b5eb (ale 10f45140: 6f6c4078 696c2d67 2d78756e 612d7068   x@log-linux-hp-a 10f45150: 7a6e6f67 20296c61 63636728 72657620   gonzal) (gcc ver 10f45160: 6e6f6973 392e3420 2820312e 29434347   sion 4.9.1 (GCC) 10f45170: 23202920 4d532031 52502050 504d4545     ) #1 SMP PREEMP 10f45180: 75532054 6546206e 35312062 3a323120   T Sun Feb 15 12: 10f45190: 333a3733 45432037 30322054 00003531   37:37 CET 2015.. 10f451a0: 00000000 00000000 00400050 82000000   ........P.@..... 10f451b0: 3a555043 4d524120 50203776 65636f72   CPU: ARMv7 Proce There's more... Another method is to store the kernel log messages and kernel panics or oops into persistent storage. The Linux kernel's persistent store support (CONFIG_PSTORE) allows you to log in to the persistent memory kept across reboots. To log panic and oops messages into persistent memory, we need to configure the kernel with the CONFIG_PSTORE_RAM configuration variable, and to log kernel messages, we need to configure the kernel with CONFIG_PSTORE_CONSOLE. We then need to configure the location of the persistent storage on an unused memory location, but keep the last 1 MB of memory free. For example, we could pass the following kernel command-line arguments to reserve a 128 KB region starting at 0x30000000: ramoops.mem_address=0x30000000 ramoops.mem_size=0x200000 We would then mount the persistent storage by adding it to /etc/fstab so that it is available on the next boot as well: /etc/fstab: pstore /pstore pstore defaults 0 0 We then mount it as follows: # mkdir /pstore # mount /pstore Next, we force a reboot with the magic SysRq key: # echo b > /proc/sysrq-trigger On reboot, we will see a file inside /pstore: -r--r--r-- 1 root root 4084 Sep 16 16:24 console-ramoops This will have contents such as the following: SysRq : Resetting CPU3: stopping CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.14.0-rc4-1.0.0-wandboard-37774-g1eae [<80014a30>] (unwind_backtrace) from [<800116cc>] (show_stack+0x10/0x14) [<800116cc>] (show_stack) from [<806091f4>] (dump_stack+0x7c/0xbc) [<806091f4>] (dump_stack) from [<80013990>] (handle_IPI+0x144/0x158) [<80013990>] (handle_IPI) from [<800085c4>] (gic_handle_irq+0x58/0x5c) [<800085c4>] (gic_handle_irq) from [<80012200>] (__irq_svc+0x40/0x70) Exception stack(0xee4c1f50 to 0xee4c1f98) We should move it out of /pstore or remove it completely so that it doesn't occupy memory. Summary This article guides you through the customization of the BSP for your own product. It then explains how to debug the Linux kernel booting process. Resources for Article: Further resources on this subject: Baking Bits with Yocto Project [article] An Introduction to the Terminal [article] Linux Shell Scripting – various recipes to help you [article]
Read more
  • 0
  • 0
  • 29199

article-image-getting-ready-coffeescript
Packt
02 Apr 2015
20 min read
Save for later

Getting Ready with CoffeeScript

Packt
02 Apr 2015
20 min read
In this article by Mike Hatfield, author of the book, CoffeeScript Application Development Cookbook, we will see that JavaScript, though very successful, can be a difficult language to work with. JavaScript was designed by Brendan Eich in a mere 10 days in 1995 while working at Netscape. As a result, some might claim that JavaScript is not as well rounded as some other languages, a point well illustrated by Douglas Crockford in his book titled JavaScript: The Good Parts, O'Reilly Media. These pitfalls found in the JavaScript language led Jeremy Ashkenas to create CoffeeScript, a language that attempts to expose the good parts of JavaScript in a simple way. CoffeeScript compiles into JavaScript and helps us avoid the bad parts of JavaScript. (For more resources related to this topic, see here.) There are many reasons to use CoffeeScript as your development language of choice. Some of these reasons include: CoffeeScript helps protect us from the bad parts of JavaScript by creating function closures that isolate our code from the global namespace by reducing the curly braces and semicolon clutter and by helping tame JavaScript's notorious this keyword CoffeeScript helps us be more productive by providing features such as list comprehensions, classes with inheritance, and many others Properly written CoffeeScript also helps us write code that is more readable and can be more easily maintained As Jeremy Ashkenas says: "CoffeeScript is just JavaScript." We can use CoffeeScript when working with the large ecosystem of JavaScript libraries and frameworks on all aspects of our applications, including those listed in the following table: Part Some options User interfaces UI frameworks including jQuery, Backbone.js, AngularJS, and Kendo UI Databases Node.js drivers to access SQLite, Redis, MongoDB, and CouchDB Internal/external services Node.js with Node Package Manager (NPM) packages to create internal services and interfacing with external services Testing Unit and end-to-end testing with Jasmine, Qunit, integration testing with Zombie, and mocking with Persona Hosting Easy API and application hosting with Heroku and Windows Azure Tooling Create scripts to automate routine tasks and using Grunt Configuring your environment and tools One significant aspect to being a productive CoffeeScript developer is having a proper development environment. This environment typically consists of the following: Node.js and the NPM CoffeeScript Code editor Debugger In this recipe, we will look at installing and configuring the base components and tools necessary to develop CoffeeScript applications. Getting ready In this section, we will install the software necessary to develop applications with CoffeeScript. One of the appealing aspects of developing applications using CoffeeScript is that it is well supported on Mac, Windows, and Linux machines. To get started, you need only a PC and an Internet connection. How to do it... CoffeeScript runs on top of Node.js—the event-driven, non-blocking I/O platform built on Chrome's JavaScript runtime. If you do not have Node.js installed, you can download an installation package for your Mac OS X, Linux, and Windows machines from the start page of the Node.js website (http://nodejs.org/). To begin, install Node.js using an official prebuilt installer; it will also install the NPM. Next, we will use NPM to install CoffeeScript. Open a terminal or command window and enter the following command: npm install -g coffee-script This will install the necessary files needed to work with CoffeeScript, including the coffee command that provides an interactive Read Evaluate Print Loop (REPL)—a command to execute CoffeeScript files and a compiler to generate JavaScript. It is important to use the -g option when installing CoffeeScript, as this installs the CoffeeScript package as a global NPM module. This will add the necessary commands to our path. On some Windows machines, you might need to add the NPM binary directory to your path. You can do this by editing the environment variables and appending ;%APPDATA%npm to the end of the system's PATH variable. Configuring Sublime Text What you use to edit code can be a very personal choice, as you, like countless others, might use the tools dictated by your team or manager. Fortunately, most popular editing tools either support CoffeeScript out of the box or can be easily extended by installing add-ons, packages, or extensions. In this recipe, we will look at adding CoffeeScript support to Sublime Text and Visual Studio. Getting ready This section assumes that you have Sublime Text or Visual Studio installed. Sublime Text is a very popular text editor that is geared to working with code and projects. You can download a fully functional evaluation version from http://www.sublimetext.com. If you find it useful and decide to continue to use it, you will be encouraged to purchase a license, but there is currently no enforced time limit. How to do it... Sublime Text does not support CoffeeScript out of the box. Thankfully, a package manager exists for Sublime Text; this package manager provides access to hundreds of extension packages, including ones that provide helpful and productive tools to work with CoffeeScript. Sublime Text does not come with this package manager, but it can be easily added by following the instructions on the Package Control website at https://sublime.wbond.net/installation. With Package Control installed, you can easily install the CoffeeScript packages that are available using the Package Control option under the Preferences menu. Select the Install Package option. You can also access this command by pressing Ctrl + Shift + P, and in the command list that appears, start typing install. This will help you find the Install Package command quickly. To install the CoffeeScript package, open the Install Package window and enter CoffeeScript. This will display the CoffeeScript-related packages. We will use the Better CoffeeScript package: As you can see, the CoffeeScript package includes syntax highlighting, commands, shortcuts, snippets, and compilation. How it works... In this section, we will explain the different keyboard shortcuts and code snippets available with the Better CoffeeScript package for Sublime. Commands You can run the desired command by entering the command into the Sublime command pallet or by pressing the related keyboard shortcut. Remember to press Ctrl + Shift + P to display the command pallet window. Some useful CoffeeScript commands include the following: Command Keyboard shortcut Description Coffee: Check Syntax Alt + Shift + S This checks the syntax of the file you are editing or the currently selected code. The result will display in the status bar at the bottom. Coffee: Compile File Alt + Shift + C This compiles the file being edited into JavaScript. Coffee: Run Script Alt + Shift + R This executes the selected code and displays a buffer of the output. The keyboard shortcuts are associated with the file type. If you are editing a new CoffeeScript file that has not been saved yet, you can specify the file type by choosing CoffeeScript in the list of file types in the bottom-left corner of the screen. Snippets Snippets allow you to use short tokens that are recognized by Sublime Text. When you enter the code and press the Tab key, Sublime Text will automatically expand the snippet into the full form. Some useful CoffeeScript code snippets include the following: Token Expands to log[Tab] console.log cla class Name constructor: (arguments) ->    # ... forin for i in array # ... if if condition # ... ifel if condition # ... else # ... swi switch object when value    # ... try try # ... catch e # ... The snippets are associated with the file type. If you are editing a new CoffeeScript file that has not been saved yet, you can specify the file type by selecting CoffeeScript in the list of file types in the bottom-left corner of the screen. Configuring Visual Studio In this recipe, we will demonstrate how to add CoffeeScript support to Visual Studio. Getting ready If you are on the Windows platform, you can use Microsoft's Visual Studio software. You can download Microsoft's free Express edition (Express 2013 for Web) from http://www.microsoft.com/express. How to do it... If you are a Visual Studio user, Version 2010 and above can work quite effectively with CoffeeScript through the use of Visual Studio extensions. If you are doing any form of web development with Visual Studio, the Web Essentials extension is a must-have. To install Web Essentials, perform the following steps: Launch Visual Studio. Click on the Tools menu and select the Extensions and Updates menu option. This will display the Extensions and Updates window (shown in the next screenshot). Select Online in the tree on the left-hand side to display the most popular downloads. Select Web Essentials 2012 from the list of available packages and then click on the Download button. This will download the package and install it automatically. Once the installation is finished, restart Visual Studio by clicking on the Restart Now button. You will likely find Web Essentials 2012 ranked highly in the list of Most Popular packages. If you do not see it, you can search for Web Essentials using the Search box in the top-right corner of the window. Once installed, the Web Essentials package provides many web development productivity features, including CSS helpers, tools to work with Less CSS, enhancements to work with JavaScript, and, of course, a set of CoffeeScript helpers. To add a new CoffeeScript file to your project, you can navigate to File | New Item or press Ctrl + Shift + A. This will display the Add New Item dialog, as seen in the following screenshot. Under the Web templates, you will see a new CoffeeScript File option. Select this option and give it a filename, as shown here: When we have our CoffeeScript file open, Web Essentials will display the file in a split-screen editor. We can edit our code in the left-hand pane, while Web Essentials displays a live preview of the JavaScript code that will be generated for us. The Web Essentials CoffeeScript compiler will create two JavaScript files each time we save our CoffeeScript file: a basic JavaScript file and a minified version. For example, if we save a CoffeeScript file named employee.coffee, the compiler will create employee.js and employee.min.js files. Though I have only described two editors to work with CoffeeScript files, there are CoffeeScript packages and plugins for most popular text editors, including Emacs, Vim, TextMate, and WebMatrix. A quick dive into CoffeeScript In this recipe, we will take a quick look at the CoffeeScript language and command line. How to do it... CoffeeScript is a highly expressive programming language that does away with much of the ceremony required by JavaScript. It uses whitespace to define blocks of code and provides shortcuts for many of the programming constructs found in JavaScript. For example, we can declare variables and functions without the var keyword: firstName = 'Mike' We can define functions using the following syntax: multiply = (a, b) -> a * b Here, we defined a function named multiply. It takes two arguments, a and b. Inside the function, we multiplied the two values. Note that there is no return statement. CoffeeScript will always return the value of the last expression that is evaluated inside a function. The preceding function is equivalent to the following JavaScript snippet: var multiply = function(a, b) { return a * b; }; It's worth noting that the CoffeeScript code is only 28 characters long, whereas the JavaScript code is 50 characters long; that's 44 percent less code. We can call our multiply function in the following way: result = multiply 4, 7 In CoffeeScript, using parenthesis is optional when calling a function with parameters, as you can see in our function call. However, note that parenthesis are required when executing a function without parameters, as shown in the following example: displayGreeting = -> console.log 'Hello, world!' displayGreeting() In this example, we must call the displayGreeting() function with parenthesis. You might also wish to use parenthesis to make your code more readable. Just because they are optional, it doesn't mean you should sacrifice the readability of your code to save a couple of keystrokes. For example, in the following code, we used parenthesis even though they are not required: $('div.menu-item').removeClass 'selected' Like functions, we can define JavaScript literal objects without the need for curly braces, as seen in the following employee object: employee = firstName: 'Mike' lastName: 'Hatfield' salesYtd: 13204.65 Notice that in our object definition, we also did not need to use a comma to separate our properties. CoffeeScript supports the common if conditional as well as an unless conditional inspired by the Ruby language. Like Ruby, CoffeeScript also provides English keywords for logical operations such as is, isnt, or, and and. The following example demonstrates the use of these keywords: isEven = (value) -> if value % 2 is 0    'is' else    'is not'   console.log '3 ' + isEven(3) + ' even' In the preceding code, we have an if statement to determine whether a value is even or not. If the value is even, the remainder of value % 2 will be 0. We used the is keyword to make this determination. JavaScript has a nasty behavior when determining equality between two values. In other languages, the double equal sign is used, such as value == 0. In JavaScript, the double equal operator will use type coercion when making this determination. This means that 0 == '0'; in fact, 0 == '' is also true. CoffeeScript avoids this using JavaScript's triple equals (===) operator. This evaluation compares value and type such that 0 === '0' will be false. We can use if and unless as expression modifiers as well. They allow us to tack if and unless at the end of a statement to make simple one-liners. For example, we can so something like the following: console.log 'Value is even' if value % 2 is 0 Alternatively, we can have something like this: console.log 'Value is odd' unless value % 2 is 0 We can also use the if...then combination for a one-liner if statement, as shown in the following code: if value % 2 is 0 then console.log 'Value is even' CoffeeScript has a switch control statement that performs certain actions based on a list of possible values. The following lines of code show a simple switch statement with four branching conditions: switch task when 1    console.log 'Case 1' when 2    console.log 'Case 2' when 3, 4, 5    console.log 'Case 3, 4, 5' else    console.log 'Default case' In this sample, if the value of a task is 1, case 1 will be displayed. If the value of a task is 3, 4, or 5, then case 3, 4, or 5 is displayed, respectively. If there are no matching values, we can use an optional else condition to handle any exceptions. If your switch statements have short operations, you can turn them into one-liners, as shown in the following code: switch value when 1 then console.log 'Case 1' when 2 then console.log 'Case 2' when 3, 4, 5 then console.log 'Case 3, 4, 5' else console.log 'Default case' CoffeeScript provides a number of syntactic shortcuts to help us be more productive while writing more expressive code. Some people have claimed that this can sometimes make our applications more difficult to read, which will, in turn, make our code less maintainable. The key to highly readable and maintainable code is to use a consistent style when coding. I recommend that you follow the guidance provided by Polar in their CoffeeScript style guide at http://github.com/polarmobile/coffeescript-style-guide. There's more... With CoffeeScript installed, you can use the coffee command-line utility to execute CoffeeScript files, compile CoffeeScript files into JavaScript, or run an interactive CoffeeScript command shell. In this section, we will look at the various options available when using the CoffeeScript command-line utility. We can see a list of available commands by executing the following command in a command or terminal window: coffee --help This will produce the following output: As you can see, the coffee command-line utility provides a number of options. Of these, the most common ones include the following: Option Argument Example Description None None coffee This launches the REPL-interactive shell. None Filename coffee sample.coffee This command will execute the CoffeeScript file. -c, --compile Filename coffee -c sample.coffee This command will compile the CoffeeScript file into a JavaScript file with the same base name,; sample.js, as in our example. -i, --interactive   coffee -i This command will also launch the REPL-interactive shell. -m, --map Filename coffee--m sample.coffee This command generates a source map with the same base name, sample.js.map, as in our example. -p, --print Filename coffee -p sample.coffee This command will display the compiled output or compile errors to the terminal window. -v, --version None coffee -v This command will display the correct version of CoffeeScript. -w, --watch Filename coffee -w -c sample.coffee This command will watch for file changes, and with each change, the requested action will be performed. In our example, our sample.coffee file will be compiled each time we save it. The CoffeeScript REPL As we have been, CoffeeScript has an interactive shell that allows us to execute CoffeeScript commands. In this section, we will learn how to use the REPL shell. The REPL shell can be an excellent way to get familiar with CoffeeScript. To launch the CoffeeScript REPL, open a command window and execute the coffee command. This will start the interactive shell and display the following prompt: For example, if we enter the expression x = 4 and press return, we would see what is shownin the following screenshot In the coffee> prompt, we can assign values to variables, create functions, and evaluate results. When we enter an expression and press the return key, it is immediately evaluated and the value is displayed. For example, if we enter the expression x = 4 and press return, we would see what is shown in the following screenshot: This did two things. First, it created a new variable named x and assigned the value of 4 to it. Second, it displayed the result of the command. Next, enter timesSeven = (value) -> value * 7 and press return: You can see that the result of this line was the creation of a new function named timesSeven(). We can call our new function now: By default, the REPL shell will evaluate each expression when you press the return key. What if we want to create a function or expression that spans multiple lines? We can enter the REPL multiline mode by pressing Ctrl + V. This will change our coffee> prompt to a ------> prompt. This allows us to enter an expression that spans multiple lines, such as the following function: When we are finished with our multiline expression, press Ctrl + V again to have the expression evaluated. We can then call our new function: The CoffeeScript REPL offers some handy helpers such as expression history and tab completion. Pressing the up arrow key on your keyboard will circulate through the expressions we previously entered. Using the Tab key will autocomplete our function or variable name. For example, with the isEvenOrOdd() function, we can enter isEven and press Tab to have the REPL complete the function name for us. Debugging CoffeeScript using source maps If you have spent any time in the JavaScript community, you would have, no doubt, seen some discussions or rants regarding the weak debugging story for CoffeeScript. In fact, this is often a top argument some give for not using CoffeeScript at all. In this recipe, we will examine how to debug our CoffeeScript application using source maps. Getting ready The problem in debugging CoffeeScript stems from the fact that CoffeeScript compiles into JavaScript which is what the browser executes. If an error arises, the line that has caused the error sometimes cannot be traced back to the CoffeeScript source file very easily. Also, the error message is sometimes confusing, making troubleshooting that much more difficult. Recent developments in the web development community have helped improve the debugging experience for CoffeeScript by making use of a concept known as a source map. In this section, we will demonstrate how to generate and use source maps to help make our CoffeeScript debugging easier. To use source maps, you need only a base installation of CoffeeScript. How to do it... You can generate a source map for your CoffeeScript code using the -m option on the CoffeeScript command: coffee -m -c employee.coffee How it works... Source maps provide information used by browsers such as Google Chrome that tell the browser how to map a line from the compiled JavaScript code back to its origin in the CoffeeScript file. Source maps allow you to place breakpoints in your CoffeeScript file and analyze variables and execute functions in your CoffeeScript module. This creates a JavaScript file called employee.js and a source map called employee.js.map. If you look at the last line of the generated employee.js file, you will see the reference to the source map: //# sourceMappingURL=employee.js.map Google Chrome uses this JavaScript comment to load the source map. The following screenshot demonstrates an active breakpoint and console in Goggle Chrome: Debugging CoffeeScript using Node Inspector Source maps and Chrome's developer tools can help troubleshoot our CoffeeScript that is destined for the Web. In this recipe, we will demonstrate how to debug CoffeeScript that is designed to run on the server. Getting ready Begin by installing the Node Inspector NPM module with the following command: npm install -g node-inspector How to do it... To use Node Inspector, we will use the coffee command to compile the CoffeeScript code we wish to debug and generate the source map. In our example, we will use the following simple source code in a file named counting.coffee: for i in [1..10] if i % 2 is 0    console.log "#{i} is even!" else    console.log "#{i} is odd!" To use Node Inspector, we will compile our file and use the source map parameter with the following command: coffee -c -m counting.coffee Next, we will launch Node Inspector with the following command: node-debug counting.js How it works... When we run Node Inspector, it does two things. First, it launches the Node debugger. This is a debugging service that allows us to step through code, hit line breaks, and evaluate variables. This is a built-in service that comes with Node. Second, it launches an HTTP handler and opens a browser that allows us to use Chrome's built-in debugging tools to use break points, step over and into code, and evaluate variables. Node Inspector works well using source maps. This allows us to see our native CoffeeScript code and is an effective tool to debug server-side code. The following screenshot displays our Chrome window with an active break point. In the local variables tool window on the right-hand side, you can see that the current value of i is 2: The highlighted line in the preceding screenshot depicts the log message. Summary This article introduced CoffeeScript and lays the foundation to use CoffeeScript to develop all aspects of modern cloud-based applications. Resources for Article: Further resources on this subject: Writing Your First Lines of CoffeeScript [article] Why CoffeeScript? [article] ASP.Net Site Performance: Improving JavaScript Loading [article]
Read more
  • 0
  • 0
  • 4474
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-machine-learning-using-spark-mllib
Packt
01 Apr 2015
22 min read
Save for later

Machine Learning Using Spark MLlib

Packt
01 Apr 2015
22 min read
This Spark machine learning tutorial is by Krishna Sankar, the author of Fast Data Processing with Spark Second Edition. One of the major attractions of Spark is the ability to scale computation massively, and that is exactly what you need for machine learning algorithms. But the caveat is that all machine learning algorithms cannot be effectively parallelized. Each algorithm has its own challenges for parallelization, whether it is task parallelism or data parallelism. Having said that, Spark is becoming the de-facto platform for building machine learning algorithms and applications. For example, Apache Mahout is moving away from Hadoop MapReduce and implementing the algorithms in Spark (see the first reference at the end of this article). The developers working on the Spark MLlib are implementing more and more machine algorithms in a scalable and concise manner in the Spark framework. For the latest information on this, you can refer to the Spark site at https://spark.apache.org/docs/latest/mllib-guide.html, which is the authoritative source. This article covers the following machine learning algorithms: Basic statistics Linear regression Classification Clustering Recommendations The Spark machine learning algorithm table The Spark machine learning algorithms implemented in Spark 1.1.0 org.apache.spark.mllib for Scala and Java, and in pyspark.mllib for Python is shown in the following table: Algorithm Feature Notes Basic statistics Summary statistics Mean, variance, count, max, min, and numNonZeros   Correlations Spearman and Pearson correlation   Stratified sampling sampleBykey, sampleByKeyExact—With and without replacement   Hypothesis testing Pearson's chi-squared goodness of fit test   Random data generation RandomRDDs Normal, Poisson, and so on Regression Linear models Linear regression—least square, Lasso, and ridge regression Classification Binary classification Logistic regression, SVM, decision trees, and naïve Bayes   Multi-class classification Decision trees, naïve Bayes, and so on Recommendation Collaborative filtering Alternating least squares Clustering k-means   Dimensionality reduction SVD PCA   Feature extraction TF-IDF Word2Vec StandardScaler Normalizer   Optimization SGD L-BFGS   Spark MLlib examples Now, let's look at how to use the algorithms. Naturally, we need interesting datasets to implement the algorithms; we will use appropriate datasets for the algorithms shown in the next section. The code and data files are available in the GitHub repository at https://github.com/xsankar/fdps-vii. We'll keep it updated with corrections. Basic statistics Let's read the car mileage data into an RDD and then compute some basic statistics. We will use a simple parse class to parse a line of data. This will work if you know the type and the structure of your CSV file. We will use this technique for the examples in this article: import org.apache.spark.SparkContext import org.apache.spark.mllib.stat. {MultivariateStatisticalSummary, Statistics} import org.apache.spark.mllib.linalg.Vector import org.apache.spark.mllib.linalg.Vectors import org.apache.spark.rdd.RDD   object MLlib01 { // def getCurrentDirectory = new java.io.File( "." ).getCanonicalPath // def parseCarData(inpLine : String) : Array[Double] = {    val values = inpLine.split(',')    val mpg = values(0).toDouble    val displacement = values(1).toDouble    val hp = values(2).toInt    val torque = values(3).toInt    val CRatio = values(4).toDouble    val RARatio = values(5).toDouble    val CarbBarrells = values(6).toInt    val NoOfSpeed = values(7).toInt    val length = values(8).toDouble    val width = values(9).toDouble    val weight = values(10).toDouble    val automatic = values(11).toInt    return Array(mpg,displacement,hp,    torque,CRatio,RARatio,CarbBarrells,    NoOfSpeed,length,width,weight,automatic) } // def main(args: Array[String]) {    println(getCurrentDirectory)    val sc = new SparkContext("local","Chapter 9")    println(s"Running Spark Version ${sc.version}")    //    val dataFile = sc.textFile("/Users/ksankar/fdps-vii/data/car-     milage-no-hdr.csv")    val carRDD = dataFile.map(line => parseCarData(line))    //    // Let us find summary statistics    //    val vectors: RDD[Vector] = carRDD.map(v => Vectors.dense(v))    val summary = Statistics.colStats(vectors)    carRDD.foreach(ln=> {ln.foreach(no => print("%6.2f | "     .format(no))); println()})    print("Max :");summary.max.toArray.foreach(m => print("%5.1f |     ".format(m)));println    print("Min :");summary.min.toArray.foreach(m => print("%5.1f |     ".format(m)));println    print("Mean :");summary.mean.toArray.foreach(m => print("%5.1f     | ".format(m)));println    } } This program will produce the following output: Let's also run some correlations, as shown here: // // correlations // val hp = vectors.map(x => x(2)) val weight = vectors.map(x => x(10)) var corP = Statistics.corr(hp,weight,"pearson") // default println("hp to weight : Pearson Correlation = %2.4f".format(corP)) var corS = Statistics.corr(hp,weight,"spearman") // Need to   specify println("hp to weight : Spearman Correlation = %2.4f" .format(corS)) // val raRatio = vectors.map(x => x(5)) val width = vectors.map(x => x(9)) corP = Statistics.corr(raRatio,width,"pearson") // default println("raRatio to width : Pearson Correlation = %2.4f" .format(corP)) corS = Statistics.corr(raRatio,width,"spearman") // Need to   specify println("raRatio to width : Spearman Correlation = %2.4f" .format(corS)) // This will produce interesting results as shown in the next screenshot: While this might seem too much work to calculate the correlation of a tiny dataset, remember that this will scale to datasets consisting of 1,000,000 rows or even a billion rows! Linear regression Linear regression takes a little more work than statistics. We need the LabeledPoint class as well as a few more parameters such as the learning rate, that is, the step size. We will also split the dataset into training and test, as shown here:    //    // def carDataToLP(inpArray : Array[Double]) : LabeledPoint = {    return new LabeledPoint( inpArray(0),Vectors.dense (       inpArray(1), inpArray(2), inpArray(3),       inpArray(4), inpArray(5), inpArray(6), inpArray(7),       inpArray(8), inpArray(9), inpArray(10), inpArray(11) ) )    } // Linear Regression    //    val carRDDLP = carRDD.map(x => carDataToLP(x)) // create a     labeled point RDD    println(carRDDLP.count())    println(carRDDLP.first().label)    println(carRDDLP.first().features)    //    // Let us split the data set into training & test set using a     very simple filter    //    val carRDDLPTrain = carRDDLP.filter( x => x.features(9) <=     4000)    val carRDDLPTest = carRDDLP.filter( x => x.features(9) > 4000)    println("Training Set : " + "%3d".format     (carRDDLPTrain.count()))    println("Training Set : " + "%3d".format(carRDDLPTest.count()))    //    // Train a Linear Regression Model    // numIterations = 100, stepsize = 0.000000001    // without such a small step size the algorithm will diverge    //    val mdlLR = LinearRegressionWithSGD.train     (carRDDLPTrain,100,0.000000001)    println(mdlLR.intercept) // Intercept is turned off when using     LinearRegressionSGD object, so intercept will always be 0 for     this code      println(mdlLR.weights)    //    // Now let us use the model to predict our test set    //    val valuesAndPreds = carRDDLPTest.map(p => (p.label,     mdlLR.predict(p.features)))    val mse = valuesAndPreds.map( vp => math.pow( (vp._1 - vp._2),2     ) ).        reduce(_+_) / valuesAndPreds.count()    println("Mean Squared Error     = " + "%6.3f".format(mse))    println("Root Mean Squared Error = " + "%6.3f"     .format(math.sqrt(mse)))    // Let us print what the model predicted    valuesAndPreds.take(20).foreach(m => println("%5.1f | %5.1f |"     .format(m._1,m._2))) The run result will be as expected, as shown in the next screenshot: The prediction is not that impressive. There are a couple of reasons for this. There might be quadratic effects; some of the variables might be correlated (for example, length, width, and weight, and so we might not need all three to predict the mpg value). Finally, we might not need all the 10 features anyways. I leave it to you to try with different combinations of features. (In the parseCarData function, take only a subset of the variables; for example take hp, weight, and number of speed and see which combination minimizes the mse value.) Classification Classification is very similar to linear regression. The algorithms take labeled points, and the train process has various parameters to tweak the algorithm to fit the needs of an application. The returned model can be used to predict the class of a labeled point. Here is a quick example using the titanic dataset: For our example, we will keep the same structure as the linear regression example. First, we will parse the full dataset line and then later keep it simple by creating a labeled point with a set of selected features, as shown in the following code: import org.apache.spark.SparkContext import org.apache.spark.mllib.regression.LabeledPoint import org.apache.spark.mllib.linalg.Vectors import org.apache.spark.mllib.tree.DecisionTree   object Chapter0802 { // def getCurrentDirectory = new java.io.File( "."     ).getCanonicalPath // // 0 pclass,1 survived,2 l.name,3.f.name, 4 sex,5 age,6 sibsp,7       parch,8 ticket,9 fare,10 cabin, // 11 embarked,12 boat,13 body,14 home.dest // def str2Double(x: String) : Double = {    try {      x.toDouble    } catch {      case e: Exception => 0.0    } } // def parsePassengerDataToLP(inpLine : String) : LabeledPoint = {    val values = inpLine.split(',')    //println(values)    //println(values.length)    //    val pclass = str2Double(values(0))    val survived = str2Double(values(1))    // skip last name, first name    var sex = 0    if (values(4) == "male") {      sex = 1    }    var age = 0.0 // a better choice would be the average of all       ages    age = str2Double(values(5))    //    var sibsp = 0.0    age = str2Double(values(6))    //    var parch = 0.0    age = str2Double(values(7))    //    var fare = 0.0    fare = str2Double(values(9))    return new LabeledPoint(survived,Vectors.dense     (pclass,sex,age,sibsp,parch,fare)) } Now that we have setup the routines to parse the data, let's dive into the main program: // def main(args: Array[String]): Unit = {    println(getCurrentDirectory)    val sc = new SparkContext("local","Chapter 8")    println(s"Running Spark Version ${sc.version}")    //    val dataFile = sc.textFile("/Users/ksankar/bdtc-2014     /titanic/titanic3_01.csv")    val titanicRDDLP = dataFile.map(_.trim).filter( _.length > 1).      map(line => parsePassengerDataToLP(line))    //    println(titanicRDDLP.count())    //titanicRDDLP.foreach(println)    //    println(titanicRDDLP.first().label)    println(titanicRDDLP.first().features)    //    val categoricalFeaturesInfo = Map[Int, Int]()    val mdlTree = DecisionTree.trainClassifier(titanicRDDLP, 2, //       numClasses        categoricalFeaturesInfo, // all features are continuous        "gini", // impurity        5, // Maxdepth        32) //maxBins    //    println(mdlTree.depth)    println(mdlTree) The tree is interesting to inspect. Check it out here:    //    // Let us predict on the dataset and see how well it works.    // In the real world, we should split the data to train & test       and then predict the test data:    //    val predictions = mdlTree.predict(titanicRDDLP.     map(x=>x.features))    val labelsAndPreds = titanicRDDLP.     map(x=>x.label).zip(predictions)    //    val mse = labelsAndPreds.map( vp => math.pow( (vp._1 -       vp._2),2 ) ).        reduce(_+_) / labelsAndPreds.count()    println("Mean Squared Error = " + "%6f".format(mse))    //    // labelsAndPreds.foreach(println)    //    val correctVals = labelsAndPreds.aggregate(0.0)((x, rec) => x       + (rec._1 == rec._2).compare(false), _ + _)    val accuracy = correctVals/labelsAndPreds.count()    println("Accuracy = " + "%3.2f%%".format(accuracy*100))    //    println("*** Done ***") } } The result obtained when you run the program is as expected. The printout of the tree is interesting, as shown here: Running Spark Version 1.1.1 14/11/28 18:41:27 INFO MemoryStore: ensureFreeSpace(163705) called with curMem=0, maxMem=2061647216 [..] 14/11/28 18:41:27 INFO SparkContext: Job finished: count at Chapter0802.scala:56, took 0.260993 s 1309 14/11/28 18:41:27 INFO SparkContext: Starting job: first at Chapter0802.scala:59 [..] 14/11/28 18:41:27 INFO SparkContext: Job finished: first at Chapter0802.scala:59, took 0.016479 s 1.0 14/11/28 18:41:27 INFO SparkContext: Starting job: first at Chapter0802.scala:60 [..] 14/11/28 18:41:27 INFO SparkContext: Job finished: first at Chapter0802.scala:60, took 0.014408 s [1.0,0.0,0.0,0.0,0.0,211.3375] 14/11/28 18:41:27 INFO SparkContext: Starting job: take at DecisionTreeMetadata.scala:66 [..] 14/11/28 18:41:28 INFO DecisionTree: Internal timing for DecisionTree: 14/11/28 18:41:28 INFO DecisionTree:   init: 0.36408 total: 0.95518 extractNodeInfo: 7.3E-4 findSplitsBins: 0.249814 extractInfoForLowerLevels: 7.74E-4 findBestSplits: 0.565394 chooseSplits: 0.201012 aggregation: 0.362411 5 DecisionTreeModel classifier If (feature 1 <= 0.0)    If (feature 0 <= 2.0)    If (feature 5 <= 26.0)      If (feature 2 <= 1.0)      If (feature 0 <= 1.0)        Predict: 1.0      Else (feature 0 > 1.0)        Predict: 1.0      Else (feature 2 > 1.0)      Predict: 1.0    Else (feature 5 > 26.0)      If (feature 2 <= 1.0)      If (feature 5 <= 38.0021)        Predict: 1.0      Else (feature 5 > 38.0021)        Predict: 1.0      Else (feature 2 > 1.0)      If (feature 5 <= 79.42500000000001)        Predict: 1.0      Else (feature 5 > 79.42500000000001)        Predict: 1.0    Else (feature 0 > 2.0)    If (feature 5 <= 25.4667)      If (feature 5 <= 7.2292)      If (feature 5 <= 7.05)        Predict: 1.0      Else (feature 5 > 7.05)        Predict: 1.0      Else (feature 5 > 7.2292)      If (feature 5 <= 15.5646)        Predict: 0.0      Else (feature 5 > 15.5646)        Predict: 1.0    Else (feature 5 > 25.4667)      If (feature 5 <= 38.0021)      If (feature 5 <= 30.6958)        Predict: 0.0      Else (feature 5 > 30.6958)        Predict: 0.0      Else (feature 5 > 38.0021)      Predict: 0.0 Else (feature 1 > 0.0)    If (feature 0 <= 1.0)    If (feature 5 <= 26.0)      If (feature 5 <= 7.05)      If (feature 5 <= 0.0)        Predict: 0.0      Else (feature 5 > 0.0)        Predict: 0.0      Else (feature 5 > 7.05)      Predict: 0.0    Else (feature 5 > 26.0)      If (feature 5 <= 30.6958)      If (feature 2 <= 0.0)        Predict: 0.0      Else (feature 2 > 0.0)        Predict: 0.0      Else (feature 5 > 30.6958)      If (feature 2 <= 1.0)        Predict: 0.0      Else (feature 2 > 1.0)        Predict: 1.0    Else (feature 0 > 1.0)    If (feature 2 <= 0.0)      If (feature 5 <= 38.0021)      If (feature 5 <= 14.4583)        Predict: 0.0      Else (feature 5 > 14.4583)        Predict: 0.0      Else (feature 5 > 38.0021)      If (feature 0 <= 2.0)        Predict: 0.0      Else (feature 0 > 2.0)        Predict: 1.0    Else (feature 2 > 0.0)      If (feature 5 <= 26.0)      If (feature 2 <= 1.0)        Predict: 0.0      Else (feature 2 > 1.0)        Predict: 0.0      Else (feature 5 > 26.0)      If (feature 0 <= 2.0)        Predict: 0.0      Else (feature 0 > 2.0)        Predict: 0.0   14/11/28 18:41:28 INFO SparkContext: Starting job: reduce at Chapter0802.scala:79 [..] 14/11/28 18:41:28 INFO SparkContext: Job finished: count at Chapter0802.scala:79, took 0.077973 s Mean Squared Error = 0.200153 14/11/28 18:41:28 INFO SparkContext: Starting job: aggregate at Chapter0802.scala:84 [..] 14/11/28 18:41:28 INFO SparkContext: Job finished: count at Chapter0802.scala:85, took 0.042592 s Accuracy = 79.98% *** Done *** In the real world, one would create a training and a test dataset and train the model on the training dataset and then predict on the test dataset. Then we can calculate the mse and minimize it on various feature combinations, some of which could also be engineered features. Clustering Spark MLlib has implemented the k-means clustering algorithm. The model training and prediction interfaces are similar to other machine learning algorithms. Let's see how it works by going through an example. Let's use a sample data that has two dimensions x and y. The plot of the points would look like the following screenshot: From the preceding graph, we can see that four clusters form one solution. Let's try with k=2 and k=4. Let's see how the Spark clustering algorithm handles this dataset and the groupings: import org.apache.spark.SparkContext import org.apache.spark.mllib.linalg.{Vector,Vectors} import org.apache.spark.mllib.clustering.KMeans   object Chapter0803 { def parsePoints(inpLine : String) : Vector = {    val values = inpLine.split(',')    val x = values(0).toInt    val y = values(1).toInt    return Vectors.dense(x,y) } //   def main(args: Array[String]): Unit = {    val sc = new SparkContext("local","Chapter 8")    println(s"Running Spark Version ${sc.version}")    //    val dataFile = sc.textFile("/Users/ksankar/bdtc-2014/cluster-     points/cluster-points.csv")    val points = dataFile.map(_.trim).filter( _.length > 1).     map(line => parsePoints(line))    //  println(points.count())    //    var numClusters = 2    val numIterations = 20    var mdlKMeans = KMeans.train(points, numClusters,       numIterations)    //    println(mdlKMeans.clusterCenters)    //    var clusterPred = points.map(x=>mdlKMeans.predict(x))    var clusterMap = points.zip(clusterPred)    //    clusterMap.foreach(println)    //    clusterMap.saveAsTextFile("/Users/ksankar/bdtc-2014/cluster-     points/2-cluster.csv")    //    // Now let us try 4 centers:    //    numClusters = 4    mdlKMeans = KMeans.train(points, numClusters, numIterations)    clusterPred = points.map(x=>mdlKMeans.predict(x))    clusterMap = points.zip(clusterPred)    clusterMap.saveAsTextFile("/Users/ksankar/bdtc-2014/cluster-     points/4-cluster.csv")    clusterMap.foreach(println) } } The results of the run would be as shown in the next screenshot (your run could give slightly different results): The k=2 graph shown in the next screenshot looks as expected: With k=4 the results are as shown in the following screenshot: The plot shown in the following screenshot confirms that the clusters are obtained as expected. Spark does understand clustering! Bear in mind that the results could vary a little between runs because the clustering algorithm picks the centers randomly and grows from there. With k=4, the results are stable; but with k=2, there is room for partitioning the points in different ways. Try it out a few times and see the results. Recommendation The recommendation algorithms fall under five general mechanisms, namely, knowledge-based, demographic-based, content-based, collaborative filtering (item-based or user-based), and latent factor-based. Usually, the collaborative filtering is computationally intensive—Spark implements the Alternating Least Square (ALS) algorithm authored by Yehuda Koren, available at http://dl.acm.org/citation.cfm?id=1608614. It is user-based collaborative filtering using the method of learning latent factors, which can scale to a large dataset. Let's quickly use the movielens medium dataset to implement a recommendation using Spark. There are some interesting RDD transformations. Apart from that, the code is not that complex, as shown next: import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ // for implicit   conversations import org.apache.spark.mllib.recommendation.Rating import org.apache.spark.mllib.recommendation.ALS   object Chapter0804 { def parseRating1(line : String) : (Int,Int,Double,Int) = {    //println(x)    val x = line.split("::")    val userId = x(0).toInt    val movieId = x(1).toInt    val rating = x(2).toDouble    val timeStamp = x(3).toInt/10    return (userId,movieId,rating,timeStamp) } // def parseRating(x : (Int,Int,Double,Int)) : Rating = {    val userId = x._1    val movieId = x._2    val rating = x._3    val timeStamp = x._4 // ignore    return new Rating(userId,movieId,rating) } // Now that we have the parsers in place, let's focus on the main program, as shown next: def main(args: Array[String]): Unit = {    val sc = new SparkContext("local","Chapter 8")    println(s"Running Spark Version ${sc.version}")    //    val moviesFile = sc.textFile("/Users/ksankar/bdtc-     2014/movielens/medium/movies.dat")    val moviesRDD = moviesFile.map(line => line.split("::"))    println(moviesRDD.count())    //    val ratingsFile = sc.textFile("/Users/ksankar/bdtc-     2014/movielens/medium/ratings.dat")    val ratingsRDD = ratingsFile.map(line => parseRating1(line))    println(ratingsRDD.count())    //    ratingsRDD.take(5).foreach(println) // always check the RDD    //    val numRatings = ratingsRDD.count()    val numUsers = ratingsRDD.map(r => r._1).distinct().count()    val numMovies = ratingsRDD.map(r => r._2).distinct().count()    println("Got %d ratings from %d users on %d movies.".          format(numRatings, numUsers, numMovies)) Split the dataset into training, validation, and test. We can use any random dataset. But here we will use the last digit of the timestamp: val trainSet = ratingsRDD.filter(x => (x._4 % 10) < 6) .map(x=>parseRating(x))    val validationSet = ratingsRDD.filter(x => (x._4 % 10) >= 6 &       (x._4 % 10) < 8).map(x=>parseRating(x))    val testSet = ratingsRDD.filter(x => (x._4 % 10) >= 8)     .map(x=>parseRating(x))    println("Training: "+ "%d".format(trainSet.count()) +      ", validation: " + "%d".format(validationSet.count()) + ",         test: " + "%d".format(testSet.count()) + ".")    //    // Now train the model using the training set:    val rank = 10    val numIterations = 20    val mdlALS = ALS.train(trainSet,rank,numIterations)    //    // prepare validation set for prediction    //    val userMovie = validationSet.map {      case Rating(user, movie, rate) =>(user, movie)    }    //    // Predict and convert to Key-Value PairRDD    val predictions = mdlALS.predict(userMovie).map {      case Rating(user, movie, rate) => ((user, movie), rate)    }    //    println(predictions.count())    predictions.take(5).foreach(println)    //    // Now convert the validation set to PairRDD:    //    val validationPairRDD = validationSet.map(r => ((r.user,       r.product), r.rating))    println(validationPairRDD.count())    validationPairRDD.take(5).foreach(println)    println(validationPairRDD.getClass())    println(predictions.getClass())    //    // Now join the validation set with predictions.    // Then we can figure out how good our recommendations are.    // Tip:    //   Need to import org.apache.spark.SparkContext._    //   Then MappedRDD would be converted implicitly to PairRDD    //    val ratingsAndPreds = validationPairRDD.join(predictions)    println(ratingsAndPreds.count())    ratingsAndPreds.take(3).foreach(println)    //    val mse = ratingsAndPreds.map(r => {      math.pow((r._2._1 - r._2._2),2)    }).reduce(_+_) / ratingsAndPreds.count()    val rmse = math.sqrt(mse)    println("MSE = %2.5f".format(mse) + " RMSE = %2.5f"     .format(rmse))    println("** Done **") } } The run results, as shown in the next screenshot, are obtained as expected: Check the following screenshot as well: Some more information is available at: The Goodby MapReduce article from Mahout News (https://mahout.apache.org/) https://spark.apache.org/docs/latest/mllib-guide.html A Collaborative Filtering ALS paper (http://dl.acm.org/citation.cfm?id=1608614) A good presentation on decision trees (http://spark-summit.org/wp-content/uploads/2014/07/Scalable-Distributed-Decision-Trees-in-Spark-Made-Das-Sparks-Talwalkar.pdf) A recommended hands-on exercise from Spark Summit 2014 (https://databricks-training.s3.amazonaws.com/movie-recommendation-with-mllib.html) Summary In this article, we looked at the most common machine learning algorithms. Naturally, ML is a vast subject and requires lot more study, experimentation, and practical experience on interesting data science problems. Two books that are relevant to Spark Machine Learning are Packt's own books Machine Learning with Spark, Nick Pentreath, and O'Reilly's Advanced Analytics with Spark, Sandy Ryza, Uri Laserson, Sean Owen, and Josh Wills. Both are excellent books that you can refer to. Resources for Article: Further resources on this subject: Driving Visual Analyses with Automobile Data (Python) [article] The Spark programming model [article] Using the Spark Shell [article]
Read more
  • 0
  • 0
  • 7085

article-image-installing-postgresql
Packt
01 Apr 2015
16 min read
Save for later

Installing PostgreSQL

Packt
01 Apr 2015
16 min read
In this article by Hans-Jürgen Schönig, author of the book Troubleshooting PostgreSQL, we will cover what can go wrong during the installation process and what can be done to avoid those things from happening. At the end of the article, you should be able to avoid all of the pitfalls, traps, and dangers you might face during the setup process. (For more resources related to this topic, see here.) For this article, I have compiled some of the core problems that I have seen over the years, as follows: Deciding on a version during installation Memory and kernel issues Preventing problems by adding checksums to your database instance Wrong encodings and subsequent import errors Polluted template databases Killing the postmaster badly At the end of the article, you should be able to install PostgreSQL and protect yourself against the most common issues popping up immediately after installation. Deciding on a version number The first thing to work on when installing PostgreSQL is to decide on the version number. In general, a PostgreSQL version number consists of three digits. Here are some examples: 9.4.0, 9.4.1, or 9.4.2 9.3.4, 9.3.5, or 9.3.6 The last digit is the so-called minor release. When a new minor release is issued, it generally means that some bugs have been fixed (for example, some time zone changes, crashes, and so on). There will never be new features, missing functions, or changes of that sort in a minor release. The same applies to something truly important—the storage format. It won't change with a new minor release. These little facts have a wide range of consequences. As the binary format and the functionality are unchanged, you can simply upgrade your binaries, restart PostgreSQL, and enjoy your improved minor release. When the digit in the middle changes, things get a bit more complex. A changing middle digit is called a major release. It usually happens around once a year and provides you with significant new functionality. If this happens, we cannot just stop or start the database anymore to replace the binaries. If the first digit changes, something really important has happened. Examples of such important events were introductions of SQL (6.0), the Windows port (8.0), streaming replication (9.0), and so on. Technically, there is no difference between the first and the second digit—they mean the same thing to the end user. However, a migration process is needed. The question that now arises is this: if you have a choice, which version of PostgreSQL should you use? Well, in general, it is a good idea to take the latest stable release. In PostgreSQL, every version number following the design patterns I just outlined is a stable release. As of PostgreSQL 9.4, the PostgreSQL community provides fixes for versions as old as PostgreSQL 9.0. So, if you are running an older version of PostgreSQL, you can still enjoy bug fixes and so on. Methods of installing PostgreSQL Before digging into troubleshooting itself, the installation process will be outlined. The following choices are available: Installing binary packages Installing from source Installing from source is not too hard to do. However, this article will focus on installing binary packages only. Nowadays, most people (not including me) like to install PostgreSQL from binary packages because it is easier and faster. Basically, two types of binary packages are common these days: RPM (Red Hat-based) and DEB (Debian-based). Installing RPM packages Most Linux distributions include PostgreSQL. However, the shipped PostgreSQL version is somewhat ancient in many cases. Recently, I saw a Linux distribution that still featured PostgreSQL 8.4, a version already abandoned by the PostgreSQL community. Distributors tend to ship older versions to ensure that new bugs are not introduced into their systems. For high-performance production servers, outdated versions might not be the best idea, however. Clearly, for many people, it is not feasible to run long-outdated versions of PostgreSQL. Therefore, it makes sense to make use of repositories provided by the community. The Yum repository shows which distributions we can use RPMs for, at http://yum.postgresql.org/repopackages.php. Once you have found your distribution, the first thing is to install this repository information for Fedora 20 as it is shown in the next listing: yum install http://yum.postgresql.org/9.4/fedora/fedora-20-x86_64/pgdg-fedora94-9.4-1.noarch.rpm Once the repository has been added, we can install PostgreSQL: yum install postgresql94-server postgresql94-contrib /usr/pgsql-9.4/bin/postgresql94-setup initdb systemctl enable postgresql-9.4.service systemctl start postgresql-9.4.service First of all, PostgreSQL 9.4 is installed. Then a so-called database instance is created (initdb). Next, the service is enabled to make sure that it is always there after a reboot, and finally, the postgresql-9.4 service is started. The term database instance is an important concept. It basically describes an entire PostgreSQL environment (setup). A database instance is fired up when PostgreSQL is started. Databases are part of a database instance. Installing Debian packages Installing Debian packages is also not too hard. By the way, the process on Ubuntu as well as on some other similar distributions is the same as that on Debian, so you can directly use the knowledge gained from this article for other distributions. A simple file called /etc/apt/sources.list.d/pgdg.list can be created, and a line for the PostgreSQL repository (all the following steps can be done as root user or using sudo) can be added: deb http://apt.postgresql.org/pub/repos/apt/ YOUR_DEBIAN_VERSION_HERE-pgdg main So, in the case of Debian Wheezy, the following line would be useful: deb http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg main Once we have added the repository, we can import the signing key: $# wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - OK Voilà! Things are mostly done. In the next step, the repository information can be updated: apt-get update Once this has been done successfully, it is time to install PostgreSQL: apt-get install "postgresql-9.4" If no error is issued by the operating system, it means you have successfully installed PostgreSQL. The beauty here is that PostgreSQL will fire up automatically after a restart. A simple database instance has also been created for you. If everything has worked as expected, you can give it a try and log in to the database: root@chantal:~# su - postgres $ psql postgres psql (9.4.1) Type "help" for help. postgres=# Memory and kernel issues After this brief introduction to installing PostgreSQL, it is time to focus on some of the most common problems. Fixing memory issues Some of the most important issues are related to the kernel and memory. Up to version 9.2, PostgreSQL was using the classical system V shared memory to cache data, store locks, and so on. Since PostgreSQL 9.3, things have changed, solving most issues people had been facing during installation. However, in PostgreSQL 9.2 or before, you might have faced the following error message: FATAL: Could not create shared memory segment DETAIL: Failed system call was shmget (key=5432001, size=1122263040, 03600) HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 1122263040 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections. If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for. The PostgreSQL documentation contains more information about shared memory configuration. If you are facing a message like this, it means that the kernel does not provide you with enough shared memory to satisfy your needs. Where does this need for shared memory come from? Back in the old days, PostgreSQL stored a lot of stuff, such as the I/O cache (shared_buffers, locks, autovacuum-related information and a lot more), in the shared memory. Traditionally, most Linux distributions have had a tight grip on the memory, and they don't issue large shared memory segments; for example, Red Hat has long limited the maximum amount of shared memory available to applications to 32 MB. For most applications, this is not enough to run PostgreSQL in a useful way—especially not if performance does matter (and it usually does). To fix this problem, you have to adjust kernel parameters. Managing Kernel Resources of the PostgreSQL Administrator's Guide will tell you exactly why we have to adjust kernel parameters. For more information, check out the PostgreSQL documentation at http://www.postgresql.org/docs/9.4/static/kernel-resources.htm. This article describes all the kernel parameters that are relevant to PostgreSQL. Note that every operating system needs slightly different values here (for open files, semaphores, and so on). Adjusting kernel parameters for Linux In this article, parameters relevant to Linux will be covered. If shmget (previously mentioned) fails, two parameters must be changed: $ sysctl -w kernel.shmmax=17179869184 $ sysctl -w kernel.shmall=4194304 In this example, shmmax and shmall have been adjusted to 16 GB. Note that shmmax is in bytes while shmall is in 4k blocks. The kernel will now provide you with a great deal of shared memory. Also, there is more; to handle concurrency, PostgreSQL needs something called semaphores. These semaphores are also provided by the operating system. The following kernel variables are available: SEMMNI: This is the maximum number of semaphore identifiers. It should be at least ceil((max_connections + autovacuum_max_workers + 4) / 16). SEMMNS: This is the maximum number of system-wide semaphores. It should be at least ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17, and it should have room for other applications in addition to this. SEMMSL: This is the maximum number of semaphores per set. It should be at least 17. SEMMAP: This is the number of entries in the semaphore map. SEMVMX: This is the maximum value of the semaphore. It should be at least 1000. Don't change these variables unless you really have to. Changes can be made with sysctl, as was shown for the shared memory. Adjusting kernel parameters for Mac OS X If you happen to run Mac OS X and plan to run a large system, there are also some kernel parameters that need changes. Again, /etc/sysctl.conf has to be changed. Here is an example: kern.sysv.shmmax=4194304 kern.sysv.shmmin=1 kern.sysv.shmmni=32 kern.sysv.shmseg=8 kern.sysv.shmall=1024 Mac OS X is somewhat nasty to configure. The reason is that you have to set all five parameters to make this work. Otherwise, your changes will be silently ignored, and this can be really painful. In addition to that, it has to be assured that SHMMAX is an exact multiple of 4096. If it is not, trouble is near. If you want to change these parameters on the fly, recent versions of OS X provide a systcl command just like Linux. Here is how it works: sysctl -w kern.sysv.shmmax sysctl -w kern.sysv.shmmin sysctl -w kern.sysv.shmmni sysctl -w kern.sysv.shmseg sysctl -w kern.sysv.shmall Fixing other kernel-related limitations If you are planning to run a large-scale system, it can also be beneficial to raise the maximum number of open files allowed. To do that, /etc/security/limits.conf can be adapted, as shown in the next example: postgres   hard   nofile   1024 postgres   soft   nofile   1024 This example says that the postgres user can have up to 1,024 open files per session. Note that this is only important for large systems; open files won't hurt an average setup. Adding checksums to a database instance When PostgreSQL is installed, a so-called database instance is created. This step is performed by a program called initdb, which is a part of every PostgreSQL setup. Most binary packages will do this for you and you don't have to do this by hand. Why should you care then? If you happen to run a highly critical system, it could be worthwhile to add checksums to the database instance. What is the purpose of checksums? In many cases, it is assumed that crashes happen instantly—something blows up and a system fails. This is not always the case. In many scenarios, the problem starts silently. RAM may start to break, or the filesystem may start to develop slight corruption. When the problem surfaces, it may be too late. Checksums have been implemented to fight this very problem. Whenever a piece of data is written or read, the checksum is checked. If this is done, a problem can be detected as it develops. How can those checksums be enabled? All you have to do is to add -k to initdb (just change your init scripts to enable this during instance creation). Don't worry! The performance penalty of this feature can hardly be measured, so it is safe and fast to enable its functionality. Keep in mind that this feature can really help prevent problems at fairly low costs (especially when your I/O system is lousy). Preventing encoding-related issues Encoding-related problems are some of the most frequent problems that occur when people start with a fresh PostgreSQL setup. In PostgreSQL, every database in your instance has a specific encoding. One database might be en_US@UTF-8, while some other database might have been created as de_AT@UTF-8 (which denotes German as it is used in Austria). To figure out which encodings your database system is using, try to run psql -l from your Unix shell. What you will get is a list of all databases in the instance that include those encodings. So where can we actually expect trouble? Once a database has been created, many people would want to load data into the system. Let's assume that you are loading data into the aUTF-8 database. However, the data you are loading contains some ASCII characters such as ä, ö, and so on. The ASCII code for ö is 148. Binary 148 is not a valid Unicode character. In Unicode, U+00F6 is needed. Boom! Your import will fail and PostgreSQL will error out. If you are planning to load data into a new database, ensure that the encoding or character set of the data is the same as that of the database. Otherwise, you may face ugly surprises. To create a database using the correct locale, check out the syntax of CREATE DATABASE: test=# h CREATE DATABASE Command:     CREATE DATABASE Description: create a new database Syntax: CREATE DATABASE name    [ [ WITH ] [ OWNER [=] user_name ]            [ TEMPLATE [=] template ]            [ ENCODING [=] encoding ]            [ LC_COLLATE [=] lc_collate ]            [ LC_CTYPE [=] lc_ctype ]           [ TABLESPACE [=] tablespace_name ]            [ CONNECTION LIMIT [=] connlimit ] ] ENCODING and the LC* settings are used here to define the proper encoding for your new database. Avoiding template pollution It is somewhat important to understand what happens during the creation of a new database in your system. The most important point is that CREATE DATABASE (unless told otherwise) clones the template1 database, which is available in all PostgreSQL setups. This cloning has some important implications. If you have loaded a very large amount of data into template1, all of that will be copied every time you create a new database. In many cases, this is not really desirable but happens by mistake. People new to PostgreSQL sometimes put data into template1 because they don't know where else to place new tables and so on. The consequences can be disastrous. However, you can also use this common pitfall to your advantage. You can place the functions you want in all your databases in template1 (maybe for monitoring or whatever benefits). Killing the postmaster After PostgreSQL has been installed and started, many people wonder how to stop it. The most simplistic way is, of course, to use your service postgresql stop or /etc/init.d/postgresql stop init scripts. However, some administrators tend to be a bit crueler and use kill -9 to terminate PostgreSQL. In general, this is not really beneficial because it will cause some nasty side effects. Why is this so? The PostgreSQL architecture works like this: when you start PostgreSQL you are starting a process called postmaster. Whenever a new connection comes in, this postmaster forks and creates a so-called backend process (BE). This process is in charge of handling exactly one connection. In a working system, you might see hundreds of processes serving hundreds of users. The important thing here is that all of those processes are synchronized through some common chunk of memory (traditionally, shared memory, and in the more recent versions, mapped memory), and all of them have access to this chunk. What might happen if a database connection or any other process in the PostgreSQL infrastructure is killed with kill -9? A process modifying this common chunk of memory might die while making a change. The process killed cannot defend itself against the onslaught, so who can guarantee that the shared memory is not corrupted due to the interruption? This is exactly when the postmaster steps in. It ensures that one of these backend processes has died unexpectedly. To prevent the potential corruption from spreading, it kills every other database connection, goes into recovery mode, and fixes the database instance. Then new database connections are allowed again. While this makes a lot of sense, it can be quite disturbing to those users who are connected to the database system. Therefore, it is highly recommended not to use kill -9. A normal kill will be fine. Keep in mind that a kill -9 cannot corrupt your database instance, which will always start up again. However, it is pretty nasty to kick everybody out of the system just because of one process! Summary In this article we have learned how to install PostgreSQL using binary packages. Some of the most common problems and pitfalls, including encoding-related issues, checksums, and versioning were discussed. Resources for Article: Further resources on this subject: Getting Started with PostgreSQL [article] PostgreSQL Cookbook - High Availability and Replication [article] PostgreSQL – New Features [article]
Read more
  • 0
  • 0
  • 5976

article-image-woocommerce-basics
Packt
01 Apr 2015
16 min read
Save for later

WooCommerce Basics

Packt
01 Apr 2015
16 min read
In this article by Patrick Rauland, author of the book WooCommerce Cookbook, we will focus on the following topics: Installing WooCommerce Installing official WooThemes plugins Manually creating WooCommerce pages Creating a WooCommerce plugin (For more resources related to this topic, see here.) A few years ago, building an online store used to be an incredibly complex task. You had to install bulky software onto your own website and pay expensive developers a significant sum of money to customize even the simplest elements of your store. Luckily, nowadays, adding e-commerce functionality to your existing WordPress-powered website can be done by installing a single plugin. In this article, we'll go over the settings that you'll need to configure before launching your online store with WooCommerce. Most of the recipes in this article are simple to execute. We do, however, add a relatively complex recipe near the end of the article to show you how to create a plugin specifically for WooCommerce. If you're going to be customizing WooCommerce with code, it's definitely worth looking at that recipe to know the best way to customize WooCommerce without affecting other parts of your site. The recipes in this article form the very basics of setting up a store, installing plugins that enhance WooCommerce, and managing those plugins. There are recipes for official WooCommerce plugins written using WooThemes as well as a recipe for unofficial plugins. Feel free to select either one. In general, the official plugins are better supported, more up to date, and have more functionality than unofficial plugins. You could always try an unofficial plugin to see whether it meets your needs, and if it doesn't, then use an official plugin that is much more likely to meet your needs. At the end of this article, your store will be fully functional and ready to display products. Installing WooCommerce WooCommerce is a WordPress plugin, which means that you need to have WordPress running on your own server to add WooCommerce. The first step is to install WooCommerce. You could do this on an established website or a brand new website—it doesn't matter. Since e-commerce is more complex than your average plugin, there's more to the installation process than just installing the plugin. Getting ready Make sure you have the permissions necessary to install plugins on your WordPress site. The easiest way to have the correct permissions is to make sure your account on your WordPress site has the admin role. How to do it… There are two parts to this recipe. The first part is installing the plugin and the second step is adding the required pages to the site. Let's have a look at the following steps for further clarity: Log in to your WordPress site. Click on the Plugins menu. Click on the Add New menu item. These steps have been demonstrated visually in the following screenshot: Search for WooCommerce. Click on the Install Now button, as shown in the following screenshot: Once the plugin has been installed, click on the Activate Plugin button. You now have WooCommerce activated on your site, which means we're half way there. E-commerce platforms need to have certain pages (such as a cart page, a checkout page, an account page, and so on) to function. We need to add those to your site. Click on the Install WooCommerce Pages button, which appears after you've activated WooCommerce. This is demonstrated in the following screenshot: How it works… WordPress has an infrastructure that allows any WordPress site to install a plugin hosted on WordPress.org. This is a secure process that is managed by WordPress.org. Installing the WooCommerce pages allows all of the e-commerce functionality to run. Without installing the pages, WooCommerce won't know which page is the cart page or the checkout page. Once these pages are set up, we're ready to have a basic store up and running. If WordPress prompts you for FTP credentials when installing the plugin, that's likely to be a permissions issue with your web host. It is a huge pain if you have to enter FTP credentials every time you want to install or update a plugin, and it's something you should take care of. You can send this link to your web host provider so they know how to change their permissions. You can refer to http://www.chrisabernethy.com/why-wordpress-asks-connection-info/ for more information to resolve this WordPress issue. Installing official WooThemes plugins WooThemes doesn't just create the WooCommerce plugin. They also create standalone plugins and hundreds of extensions that add extra functionality to WooCommerce. The beauty of this system is that WooCommerce is very easy to use because users only add extra complexity when they need it. If you only need simple shipping options, you don't ever have to see the complex shipping settings. On the WooThemes website, you may browse for WooCommerce extensions, purchase them, and download and install them on your site. WooThemes has made the whole process very easy to maintain. They have built an updater similar to the one in WordPress, which, once configured, will allow a user to update a plugin with one click instead of having to through the whole plugin upload process again. Getting ready Make sure you have the necessary permissions to install plugins on your WordPress site. You also need to have a WooThemes product. There are several free WooThemes products including Pay with Amazon which you can find at http://www.woothemes.com/products/pay-with-amazon/. How to do it… There are two parts to this recipe. The first part is installing the plugin and the second step is adding your license for future updates. Follow these steps: Log in to http://www.woothemes.com. Click on the Downloads menu: Find the product you wish to download and click on the Download link for the product. You will see that you get a ZIP file. On your WordPress site, go the Plugins menu and click on Add New. Click on Upload Plugin. Select the file you just downloaded and click on the Install Now button. After the plugin has finished installing, click on the Activate Plugin link. You now have WooCommerce as well as a WooCommerce extension activated on your site. They're both functioning and will continue to function. You will, however, want to perform a few more steps to make sure it's easy to update your extensions: Once you have an extension activated on your site, you'll see a link in the WordPress admin: Install the WooThemes Updater plugin. Click on that link: The updater will be installed automatically. Once it is installed, you need to activate the updater. After activation, you'll see a new link in the WordPress admin: activate your product licenses. Click that link to go straight to the page where you can enter your licenses. You could also navigate to that page manually by going to Dashboard | WooThemes Helper from the menu. Keep your WordPress site open in one tab and log back in to your WooThemes account in another browser tab. On the WooThemes browser tab, go to My Licenses and you'll see a list of your products with a license key under the heading KEY: Copy the key, go back to your WordPress site, and enter it in the Licenses field. Click on the Activate Products button at the bottom of the page. The activation process can take a few seconds to complete. If you've successfully put in your key, you should see a message at the top of the screen saying so. How it works… A plugin that's not hosted on WordPress.org can't update without someone manually reuploading it. The WooThemes updater was built to make this process easier so you can press the update button and have your website do all the heavy lifting. Some websites sell official WooCommerce plugins without a license key. These sales aren't licensed and you won't be getting updates, bug fixes, or access to the support desk. With a regular website, it's important to stay up to date. However, with e-commerce, it's even more important since you'll be handling very sensitive payment information. That's why I wouldn't ever recommend using a plugin that can't update. Manually creating WooCommerce pages Every e-commerce platform will need to have some way of creating extra pages for e-commerce functionality, such as a cart page, a checkout page, an account page, and so on. WooCommerce prompts to helps you create these pages for you when you first install the plugin. So if you installed it correctly, you shouldn't have to do this. But if you were trying multiple e-commerce systems and for some reason deleted some pages, you may have to recreate those pages. How to do it… There's a very useful Tools menu in WooCommerce. It's a bit hard to find since you won't be needing it everyday, but it has some pretty useful tools if you ever need to do some troubleshooting. One of these tools is the one that allows you to recreate your WooCommerce pages. Let's have a look at how to use that tool: Log in to the WordPress admin. Click on WooCommerce | System Status: Click on Tools: Click on the Install Pages button: How it works… WooCommerce keeps track of which pages run e-commerce functionality. When you click on the Install Pages button, it checks which pages exist and if they don't exist, it will automatically create them for you. You could create them by creating new WordPress pages and then manually assigning each page with specific e-commerce functionality. You may want to do this if you already have a cart page and don't want to recreate a new cart page but just copy the content from the old page to the new page. All you want to do is tell WooCommerce which page should have the cart functionality. Let's have a look at the following manual settings: The Cart, Checkout, and Terms & Conditions page can all be set by going to WooCommerce | Settings | Checkout The My Account page can be set by going to WooCommerce | Settings | Accounts There's more... You can manually set some pages, such as the Cart and Checkout page, but you can't set subpages. WooCommerce uses a WordPress functionality called end points to create these subpages. Pages such as the Order Received page, which is displayed right after payment, can't be manually created. These endpoints are created on the fly based on the parent page. The Order Received page is part of the checkout process, so it's based on the Checkout page. Any content on the Checkout page will appear on both the Checkout page and on the Order Received page. You can't add content to the parent page without it affecting the subpage, but you can change the subpage URLs. The checkout endpoints can be configured by going to WooCommerce | Settings | Checkout | Checkout Endpoints. Creating a WooCommerce plugin Unlike a lot of hosted e-commerce solutions, WooCommerce is entirely customizable. That's one of the huge advantages for anyone who builds on open source software. If you don't like it, you can change it. At some point, you'll probably want to change something that's not on a settings page, and that's when you may want to dig into the code. Even if you don't know how to code, you may want to look this over so that when you work with a developer, you would know they're doing it the right way. Getting ready In addition to having admin access to a WordPress site, you'll also need FTP credentials so you can upload a plugin. You'll also need a text editor. Popular code editors include Sublime Text, Coda, Dreamweaver, and Atom. I personally use Atom. You could also use Notepad on a Windows machine or Text Edit on a Mac in a pinch. How to do it… We're going to be creating a plugin that interacts with WooCommerce. It will take the existing WooCommerce functionality and change it. These are the WooCommerce basics. If you build a plugin like this correctly, when WooCommerce isn't active, it won't do anything at all and won't slow down your website. Let's create a plugin by performing the following steps: Open your text editor and create a new file. Save the file as woocommerce-demo-plugin.php. In that file, add the opening PHP tag, which looks like this: <?php. On the next line, add a plugin header. This allows WordPress to recognize the file as a plugin so that it can be activated. It looks something like the following: /** * Plugin Name: WooCommerce Demo Plugin * Plugin URI: https://gist.github.com/BFTrick/3ab411e7cec43eff9769 * Description: A WooCommerce demo plugin * Author: Patrick Rauland * Author URI: http://speakinginbytes.com/ * Version: 1.0 * * This program is free software: you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation, either version 3 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program. If not, see <http://www.gnu.org/licenses/>. * */ Now that WordPress knows that your file is a plugin, it's time to add some functionality to this. The first thing a good developer does is makes sure their plugin won't conflict with another plugin. To do that, we make sure an existing class doesn't have the same name as our class. I'll be using the WC_Demo_Plugin class, but you can use any class name you want. Add the following code beneath the plugin header: if ( class_exists( 'WC_Demo_Plugin' ) ) {    return; }   class WC_Demo_Plugin {   } Our class doesn't do anything yet, but at least we've written it in such a way that it won't break another plugin. There's another good practice we should add to our plugin before we add the functionality, and that's some logic to make sure another plugin won't misuse our plugin. In the vast majority of use cases, you want to make sure there can't be two instances of your code running. In computer science, this is called the Singleton pattern. This can be controlled by tracking the instances of the plugin with a variable. Right after the WC_Demo_Plugin { line, add the following: protected static $instance = null;     /** * Return an instance of this class. * * @return object A single instance of this class. * @since 1.0 */ public static function get_instance() {    // If the single instance hasn't been set, set it now.    if ( null == self::$instance ) {        self::$instance = new self;    }      return self::$instance; } And get the plugin started by adding this right before the endif; line: add_action( 'plugins_loaded', array( 'WC_Demo_Plugin', 'get_instance' ), 0 ); At this point, we've made sure our plugin doesn't break other plugins and we've also dummy-proofed our own plugin so that we or other developers don't misuse it. Let's add just a bit more logic so that we don't run our logic unless WooCommerce is already loaded. This will make sure that we don't accidentally break something if we turn WooCommerce off temporarily. Right after the protected static $instance = null; line, add the following: /** * Initialize the plugin. * * @since 1.0 */ private function __construct() {    if ( class_exists( 'WooCommerce' ) ) {      } } And now our plugin only runs when WooCommerce is loaded. I'm guessing that at this point, you finally want it to do something, right? After we make sure WooCommerce is running, let's add some functionality. Right after the if ( class_exists( 'WooCommerce' ) ) { line, add the following code so that we add an admin notice: // print an admin notice to the screen. add_action( 'admin_notices', array( $this, 'my_admin_notice' ) ); This code will call a method called my_admin_notice, but we haven't written that yet, so it's not doing anything. Let's write that method. Have a look at the __construct method, which should now look like this: /** * Initialize the plugin. * * @since 1.0 */ private function __construct() {    if ( class_exists( 'WooCommerce' ) ) {          // print an admin notice to the screen.        add_action( 'admin_notices', array( $this, 'display_admin_notice' ) );      } } Add the following after the preceding __construct method: /** * Print an admin notice * * @since 1.0 */ public function display_admin_notice() {    ?>    <div class="updated">        <p><?php _e( 'The WooCommerce dummy plugin notice.', 'woocommerce-demo-plugin' ); ?></p>    </div>    <?php } This will print an admin notice on every single admin page. This notice includes all the messages you typically see in the WordPress admin. You could replace this admin notice method with just about any other hook in WooCommerce to provide additional customizations in other areas of WooCommerce, whether it be for shipping, the product page, the checkout process, or any other area. This plugin is the easiest way to get started with WooCommerce customizations. If you'd like to see the full code sample, you can see it at https://gist.github.com/BFTrick/3ab411e7cec43eff9769. Now that the plugin is complete, you need to upload it to your plugins folder. You can do this via the WordPress admin or more commonly via FTP. Once the plugin has been uploaded to your site, you'll need to activate the plugin just like any other WordPress plugin. The end result is a notice in the WordPress admin letting us know we did everything successfully. Whenever possible, use object-oriented code. That means using objects (like the WC_Demo_Plugin class) to encapsulate your code. It will prevent a lot of naming conflicts down the road. If you see some procedural code online, you can usually convert it to object-oriented code pretty easily. Summary In this article, you have learned the basic steps in installing WooCommerce, installing WooThemes plugins, manually creating WooCommerce pages, and creating a WooCommerce plugin. Resources for Article: Further resources on this subject: Creating Blog Content in WordPress [article] Tips and Tricks [article] Setting Up WooCommerce [article]
Read more
  • 0
  • 0
  • 11289

article-image-static-interactive-and-dynamic-dashboard
Packt
01 Apr 2015
10 min read
Save for later

From a Static to an Interactive and Dynamic Dashboard

Packt
01 Apr 2015
10 min read
In this article, by David Lai and Xavier Hacking, authors of SAP BusinessObjects Dasboards 4.1 Cookbook, we will provide developers with recipes on interactivity and look and feel of the dashboards, which will improve the dashboard user experience. We will cover the following recipes: Using the Hierarchical Table Inputting data values Displaying alerts on a map Changing the look of a chart (For more resources related to this topic, see here.) An important strength that SAP BusinessObjects Dashboards has is the amount of control it allows a developer to provide the user with. This leads to totally customized dashboards, which give users the interactivity that guides them to make the right business decisions. It is important that developers know what type of interactive tools are available so that they can utilize the power of these tools. With the right interactivity, users can retrieve information more quickly and efficiently. Using the Hierarchical Table The Hierarchical Table is a powerful component that was introduced in SAP BusinessObjects Dashboards 4.0 FP3. It allows users to connect to either a BEx query connection or an OLAP universe and take advantage of its hierarchical display and multi-selection capability. Before the Hierarchical Table was introduced, there was no way to accomplish native hierarchical display and selection without significant workarounds. Although the Hierarchical Table component is extremely powerful, please note that it can only be used with either a BEx query or an OLAP universe. It will not work on a universe based on a relational database. Getting ready Before you can take advantage of the Hierarchical Table component, you must have an OLAP universe or a BEx query connection available. In our example, we create a simple cube from the Adventureworks data warehouse, which is easily accessible from MSDN. You can download the Adventureworks data warehouse available at http://msftdbprodsamples.codeplex.com/releases/view/105902. To set up a simple cube, please follow the instructions available at http://www.accelebrate.com/library/tutorials/ssas-2008. To set up an OLAP connection to the cube, please follow the instructions available at http://wiki.scn.sap.com/wiki/display/BOBJ/Setting+up+OLAP+Microsoft+Analysis+Service+through+an+XMLA+connection+with+SSO. Finally, you will have to set up an OLAP universe that connects to the OLAP connection. Instructions for this can be found at http://scn.sap.com/docs/DOC-22026. How to do it… Create an OLAP universe query / BEx query from the Query Browser. From the Components window, go to the Selectors category and drag a Hierarchical Table component onto the dashboard canvas. Click on the Bind to Query Data button and choose the query that you created in step 1. Next, choose the dimensions and measures that you want displayed on the Hierarchical Table. By default, you must select at least one hierarchy dimension. Click on the Configure Columns button below the data binding to adjust the column widths on the Hierarchical Table. We do this because by default, SAP BusinessObjects Dashboards does not set the column widths very well when we first bind the data. On the Appearance tab, edit the number formats for each measure appropriately. For example, you can set dollar amounts as the currency with two decimal places. Next, we want to capture rows that are selected during runtime. To do this, click on the Insertion tab. For the Insertion Type, you have the option of Value or Row. For the Value insertion option, you must choose an option for Source Data, which is one of the columns in the Hierarchical Table. In our example, we will choose the Insertion Type as Row, which grabs values from all the columns. We'll need to bind the output destination. We will assume that a user can select a maximum of 30 rows. So we'll bind the output to a 30 x 3 destination range. Bind a spreadsheet table object to the destination output from step 8 to prove that our selection works. Finally, test the Hierarchical Table by entering preview mode. In the following screenshot, you can see that we can expand/collapse our Hierarchical Table, as well as make multiple selections! How it works... As you can see, the Hierarchical Table selector is a very useful component because before this component was available, we were unable to perform any form of hierarchical analysis as well as simple multi-selection. The component achieves hierarchical capabilities by taking advantage of the OLAP cube engine. There's more… Unfortunately, the Hierarchical Table selector is only available from cube sources and not a traditional data warehouse table, because it uses the OLAP cube engine to do the processing. The hierarchical capability, in our opinion, is doable with data warehouse tables as other tools allow this. So hopefully, SAP will one day upgrade the Hierarchical Table selector so that it works with your traditional data warehouse universe based on tables. Inputting data values The ability to input values into the dashboard is a very useful feature. In the following example, we have a sales forecast that changes according to an inputted number value. If we were to use a slider component for the input value, it would be more difficult for the user to select their desired input value. Another good example could be a search box to find a value on a selector which has over 100 items. This way you don't need to hunt for your value. Instead, you can just type it in. In this recipe, we will create an input textbox to control a what-if scenario. Getting ready Create a chart with its values bound to cells that will be controlled by the input textbox value. The following is an example of a sales forecast chart and its cells that are controlled by the what-if scenario: You may refer to the source file Inputting data values.xlf from the code bundle to retrieve the pre-populated data from the preceding image if you don't want to manually type everything in yourself. How to do it... Drag an Input Text object from the Text section of the Components window onto the canvas. In the properties window of the Input Text component, bind the Link to Cell as well as Destination to cell D3 from the Getting ready section. Go to the Behavior icon of the input text properties and make sure Treat All Input As Text is unchecked. The blue cell D6 from the Getting ready section that's labeled as valid value will check to make sure the input text entered by the user is valid. To do this, we use the following formula: =IF(ISNUMBER(D3),IF(AND(D3>=-20,D3<=20),D3,"INVALID"),"INVALID") The formula checks to make sure that the cell contains a number that is between -20 and 20. Now every cell in the chart binding destination will depend on D6. The binding destination cells will not add the D6 value if D6 is "INVALID". In addition, a pop up will appear saying "Input is invalid" if D6 is "INVALID". Create the pop up by dragging a Label text component onto the canvas with Input is invalid as its text. Next, go to the behavior tab and for dynamic visibility, bind it to D6 and set the Key as INVALID. How it works... In this example, we use an input value textbox to control the forecast bars on the chart. If we type 20, it will add 20 to each value in the forecast. If we type -20, it will subtract 20 from each value in the forecast. We also add a check in step 4 that determines whether the value entered is valid or not; hence the use of Excel formulas. If a value is invalid, we want to output an error to the user so that they are aware that they entered an invalid value.   Displaying alerts on a map A map on a dashboard allows us to visually identify how different regions are doing using a picture instead of a table or chart. With alerts on the map, we can provide even more value. For example, look at the following screenshot. We can see that different regions of the map can be colored differently depending on their value. This allows users to identify at a glance whether a region is doing well or poorly.   Getting ready Insert a Canadian map object into the canvas and bind data to the map. You may also refer to the data prepared in the source file, Displaying alerts on a map.xlf. How to do it... In a separate area of the spreadsheet (highlighted in yellow), set up the threshold values. Assume that all provinces have the same threshold. Go to the Alerts section of the map properties and check Enable Alerts. Select the radio button By Value. In the Alert Thresholds section, check Use a Range. Then, bind the range to the Threshold dataset in step 1. In the Color Order section, select the radio button High values are good. How it works... In this recipe, we show how to set up alerting for a map component. The way we set it up is pretty standard from steps 2 through 5. Once the alerting mechanism is set up, each province in the map will have its value associated with the alert threshold that we set up in step 1. The province will be colored red if the sales value is less than the yellow threshold. The province will be colored yellow if the sales value is greater than or equal to the yellow threshold but less than the green threshold. The province will be colored green if the sales value is greater than or equal to the green threshold. Changing the look of a chart This recipe will explain how to change the look of a chart. Particularly, it will go through each tab in the appearance icon of the chart properties. We will then make modifications and see the resulting changes. Getting ready Insert a chart object into the canvas. Prepare some data and bind it to the chart. How to do it... Click on the chart object on the canvas/object properties window to go to chart properties. In the Layout tab, uncheck Show Chart Background. In the Series tab, click on the colored box under Fill to change the color of the bar to your desired color. Then change the width of each bar; click on the Marker Size area and change it to 35. Click on the colored boxes circled in red in the Axes tab and choose dark blue as the Line Color for the horizontal and vertical axes separately. Uncheck Show Minor Gridlines to remove all the horizontal lines in between each of the major gridlines. Next, go to the Text and Color tabs, where you can make changes to all the different text areas of the chart, as shown in the following screenshot: How it works... As you can see, the default chart looks plain and the bars are skinny so it's harder to visualize things. It is a good idea to remove the chart background if there is one so that the chart blends in better. In addition, the changes to the chart colors and text provide additional aesthetics that help improve the look of the chart. Summary In this article, we learned various recipes on how to make interactive dashboards including using Hierarchical Table and alerts. Using such techniques greatly improve the look and feel of the dashboards and help create great presentations. Resources for Article: Further resources on this subject: Creating Our First Universe [article] Report Data Filtering [article] End User Transactions [article]
Read more
  • 0
  • 0
  • 2254
article-image-factor-variables-r
Packt
01 Apr 2015
7 min read
Save for later

Factor variables in R

Packt
01 Apr 2015
7 min read
This article by Jaynal Abedin and Kishor Kumar Das, authors of the book Data Manipulation with R Second Edition, will discuss factor variables in R. In any data analysis task, the majority of the time is dedicated to data cleaning and preprocessing. Sometimes, it is considered that about 80 percent of the effort is devoted to data cleaning before conducting the actual analysis. Also, in real-world data, we often work with categorical variables. A variable that takes only a limited number of distinct values is usually known as a categorical variable, and in R, it is known as a factor. Working with categorical variables in R is a bit technical, and in this article, we have tried to demystify this process of dealing with categorical variables. (For more resources related to this topic, see here.) During data analysis, the factor variable sometimes plays an important role, particularly in studying the relationship between two categorical variables. In this section, we will see some important aspects of factor manipulation. When a factor variable is first created, it stores all its levels along with the factor. But if we take any subset of that factor variable, it inherits all its levels from the original factor levels. This feature sometimes creates confusion in understanding the results. Numeric variables are convenient during statistical analysis, but sometimes, we need to create categorical (factor) variables from numeric variables. We can create a limited number of categories from a numeric variable using a series of conditional statements, but this is not an efficient way to perform this operation. In R, cut is a generic command to create factor variables from numeric variables. The split-apply-combine strategy Data manipulation is an integral part of data cleaning and analysis. For large data, it is always preferable to perform the operation within a subgroup of a dataset to speed up the process. In R, this type of data manipulation can be done with base functionality, but for large-scale data, it requires considerable amount of coding and eventually takes a longer time to process. In the case of big data, we can split the dataset, perform the manipulation or analysis, and then again combine the results into a single output. This type of split using base R is not efficient, and to overcome this limitation, Wickham developed an R package, plyr, where he efficiently implemented the split-apply-combine strategy. Often, we require similar types of operations in different subgroups of a dataset, such as group-wise summarization, standardization, and statistical modeling. This type of task requires us to break down a big problem into manageable pieces, perform operations on each piece separately, and finally combine the output of each piece into a single piece of output. To understand the split-apply-combine strategy intuitively, we can compare it with the map-reduce strategy for processing large amounts of data, recently popularized by Google. In the map-reduce strategy, the map step corresponds to split and apply and the reduce step consists of combining. The map-reduce approach is primarily designed to deal with a highly parallel environment where the work has been done by several hundreds or thousands of computers independently. The split-apply-combine strategy creates an opportunity to see the similarities of problems across subgroups that were not previously connected. This strategy can be used in many existing tools, such as the GROUP BY operation in SAS, PivotTable in MS Excel, and the SQL GROUP BY operator. The plyr package works on every type of data structure, whereas the dplyr package is designed to work only on data frames. The dplyr package offers a complete set of functions to perform every kind of data manipulation we would need in the process of analysis. These functions take a data frame as the input and also produce a data frame as the output, hence the name dplyr. There are two different types of functions in the dplyr package: single-table and aggregate. The single-table function takes a data frame as the input and an action such as subsetting a data frame, generating new columns in the data frame, or rearranging a data frame. The aggregate function takes a column as the input and produces a single value as the output, which is mostly used to summarize columns. These functions do not allow us to perform any group-wise operation, but a combination of these functions with the group_by() function allows us to implement the split-apply-combine approach. Reshaping a dataset Reshaping data is a common and tedious task in real-life data manipulation and analysis. A dataset might come with different levels of grouping, and we need to implement some reorientation to perform certain types of analyses. A dataset's layout could be long or wide. In a long layout, multiple rows represent a single subject's record, whereas in a wide layout, a single row represents a single subject's record. Statistical analysis sometimes requires wide data and sometimes long data, and in such cases, we need to be able to fluently and fluidly reshape the data to meet the requirements of statistical analysis. Data reshaping is just a rearrangement of the form of the data—it does not change the content of the dataset. In this article, we will show you different layouts of the same dataset and see how they can be transferred from one layout to another. This article mainly highlights the melt and cast paradigm of reshaping datasets, which is implemented in the reshape contributed package. Later on, this same package is reimplemented with a new name, reshape2, which is much more time and memory efficient. A single dataset can be rearranged in many different ways, but before going into rearrangement, let's look back at how we usually perceive a dataset. Whenever we think about any dataset, we think of a two-dimensional arrangement where a row represents a subject's (a subject could be a person and is typically the respondent in a survey) information for all the variables in a dataset, and a column represents the information for each characteristic for all subjects. This means that rows indicate records and columns indicate variables, characteristics, or attributes. This is the typical layout of a dataset. In this arrangement, one or more variables might play a role as an identifier, and others are measured characteristics. For the purpose of reshaping, we can group the variables into two groups: identifier variables and measured variables: The identifier variables: These help us identify the subject from whom we took information on different characteristics. Typically, identifier variables are qualitative in nature and take a limited number of unique values. In database terminology, an identifier is termed as the primary key, and this can be a single variable or a composite of multiple variables. The measured variables: These are those characteristics whose information we took from a subject of interest. These can be qualitative, quantitative, or a mixture of both. Now, beyond this typical structure of a dataset, we can think differently, where we will have only identification variables and a value. The identification variable identifies a subject along with which the measured variable the value represents. In this new paradigm, each row represents one observation of one variable. In the new paradigm, this is termed as melting and it produces molten data. The difference between this new layout of the data and the typical layout is that it now contains only the ID variable and a new column, value, which represents the value of that observation. Text processing Text data is one of the most important areas in the field of data analytics. Nowadays, we are producing a huge amount of text data through various media every day; for example, Twitter posts, blog writing, and Facebook posts are all major sources of text data. Text data can be used to retrieve information, in sentiment analysis and even entity recognition. Summary This article briefly explained the factor variables, the split-apply-combine strategy, reshaping a dataset in R, and text processing. Resources for Article: Further resources on this subject: Introduction to S4 Classes [Article] Warming Up [Article] Driving Visual Analyses with Automobile Data (Python) [Article]
Read more
  • 0
  • 0
  • 5065

article-image-optimizing-javascript-ios-hybrid-apps
Packt
01 Apr 2015
17 min read
Save for later

Optimizing JavaScript for iOS Hybrid Apps

Packt
01 Apr 2015
17 min read
In this article by Chad R. Adams, author of the book, Mastering JavaScript High Performance, we are going to take a look at the process of optimizing JavaScript for iOS web apps (also known as hybrid apps). We will take a look at some common ways of debugging and optimizing JavaScript and page performance, both in a device's web browser and a standalone app's web view. Also, we'll take a look at the Apple Web Inspector and see how to use it for iOS development. Finally, we will also gain a bit of understanding about building a hybrid app and learn the tools that help to better build JavaScript-focused apps for iOS. Moreover, we'll learn about a class that might help us further in this. We are going to learn about the following topics in the article: Getting ready for iOS development iOS hybrid development (For more resources related to this topic, see here.) Getting ready for iOS development Before starting this article with Xcode examples and using iOS Simulator, I will be displaying some native code and will use tools that haven't been covered in this course. Mobile app developments, regardless of platform, are books within themselves. When covering the build of the iOS project, I will be briefly going over the process of setting up a project and writing non-JavaScript code to get our JavaScript files into a hybrid iOS WebView for development. This is essential due to the way iOS secures its HTML5-based apps. Apps on iOS that use HTML5 can be debugged, either from a server or from an app directly, as long as the app's project is built and deployed in its debug setting on a host system (meaning the developers machine). Readers of this article are not expected to know how to build a native app from the beginning to the end. And that's completely acceptable, as you can copy-and-paste, and follow along as I go. But I will show you the code to get us to the point of testing JavaScript code, and the code used will be the smallest and the fastest possible to render your content. All of these code samples will be hosted as an Xcode project solution of some type on Packt Publishing's website, but they will also be shown here if you want to follow along, without relying on code samples. Now with that said, lets get started… iOS hybrid development Xcode is the IDE provided by Apple to develop apps for both iOS devices and desktop devices for Macintosh systems. As a JavaScript editor, it has pretty basic functions, but Xcode should be mainly used in addition to a project's toolset for JavaScript developers. It provides basic code hinting for JavaScript, HTML, and CSS, but not more than that. To install Xcode, we will need to start the installation process from the Mac App Store. Apple, in recent years, has moved its IDE to the Mac App Store for faster updates to developers and subsequently app updates for iOS and Mac applications. Installation is easy; simply log in with your Apple ID in the Mac App Store and download Xcode; you can search for it at the top or, if you look in the right rail among popular free downloads, you can find a link to the Xcode Mac App Store page. Once you reach this, click Install as shown in the following screenshot: It's important to know that, for the sake of simplicity in this article, we will not deploy an app to a device; so if you are curious about it, you will need to be actively enrolled in Apple's Developer Program. The cost is 99 dollars a year, or 299 dollars for an enterprise license that allows deployment of an app outside the control of the iOS App Store. If you're curious to learn more about deploying to a device, the code in this article will run on the device assuming that your certificates are set up on your end. For more information on this, check out Apple's iOS Developer Center documentation online at https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/AppDistributionGuide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40012582. Once it's installed, we can open up Xcode and look at the iOS Simulator; we can do this by clicking XCode, followed by Open Developer Tool, and then clicking on iOS Simulator. Upon first opening iOS Simulator, we will see what appears to be a simulation of an iOS device, shown in the next screenshot. Note that this is a simulation, not a real iOS device (even if it feels pretty close). A neat trick for JavaScript developers working with local HTML files outside an app is that they can quickly drag-and-drop an HTML file. Due to this, the simulator will open the mobile version of Safari, the built-in browser for iPhone and iPads, and render the page as it would do on an iOS device; this is pretty helpful when testing pages before deploying them to a web server. Setting up a simple iOS hybrid app JavaScript performance on a built-in hybrid application can be much slower than the same page run on the mobile version of Safari. To test this, we are going to build a very simple web browser using Apple's new programming language Swift. Swift is an iOS-ready language that JavaScript developers should feel at home with. Swift itself follows a syntax similar to JavaScript but, unlike JavaScript, variables and objects can be given types allowing for stronger, more accurate coding. In that regard, Swift follows syntax similar to what can be seen in the ECMAScript 6 and TypeScript styles of coding practice. If you are checking these newer languages out, I encourage you to check out Swift as well. Now let's create a simple web view, also known as a UIWebView, which is the class used to create a web view in an iOS app. First, let's create a new iPhone project; we are using an iPhone to keep our app simple. Open Xcode and select the Create new XCode project project; then, as shown in the following screenshot, select the Single View Application option and click the Next button. On the next view of the wizard, set the product name as JS_Performance, the language to Swift, and the device to iPhone; the organization name should autofill with your name based on your account name in the OS. The organization identifier is a reverse domain name unique identifier for our app; this can be whatever you deem appropriate. For instructional purposes, here's my setup: Once your project names are set, click the Next button and save to a folder of your choice with Git repository left unchecked. When that's done, select Main.storyboard under your Project Navigator, which is found in the left panel. We should be in the storyboard view now. Let's open the Object Library, which can be found in the lower-right panel in the subtab with an icon of a square inside a circle. Search for Web View in the Object Library in the bottom-right search bar, and then drag that to the square view that represents our iOS view. We need to consider two more things before we link up an HTML page using Swift; we need to set constraints as native iOS objects will be stretched to fit various iOS device windows. To fill the space, you can add the constraints by selecting the UIWebView object and pressing Command + Option + Shift + = on your Mac keyboard. Now you should see a blue border appear briefly around your UIWebView. Lastly, we need to connect our UIWebView to our Swift code; for this, we need to open the Assistant Editor by pressing Command + Option + Return on the keyboard. We should see ViewController.swift open up in a side panel next to our Storyboard. To link this as a code variable, right-click (or option-click the UIWebView object) and, with the button held down, drag the UIWebView to line number 12 in the ViewController.swift code in our Assistant Editor. This is shown in the following diagram: Once that's done, a popup will appear. Now leave everything the same as it comes up, but set the name to webview; this will be the variable referencing our UIWebView. With that done, save your Main.storyboard file and navigate to your ViewController.swift file. Now take a look at the Swift code shown in the following screenshot, and copy it into the project; the important part is on line 19, which contains the filename and type loaded into the web view; which in this case, this is index.html. Now obviously, we don't have an index.html file, so let's create one. Go to File and then select New followed by the New File option. Next, under iOS select Empty Application and click Next to complete the wizard. Save the file as index.html and click Create. Now open the index.html file, and type the following code into the HTML page: <br />Hello <strong>iOS</strong> Now click Run (the play button in the main iOS task bar), and we should see our HTML page running inside our own app, as shown here: That's nice work! We built an iOS app with Swift (even if it's a simple app). Let's create a structured HTML page; we will override our Hello iOS text with the HTML shown in the following screenshot: Here, we use the standard console.time function and print a message to our UIWebView page when finished; if we hit Run in Xcode, we will see the Loop Completed message on load. But how do we get our performance information? How can we get our console.timeEnd function code on line 14 on our HTML page? Using Safari web inspector for JavaScript performance Apple does provide a Web Inspector for UIWebViews, and it's the same inspector for desktop Safari. It's easy to use, but has an issue: the inspector only works on iOS Simulators and devices that have started from an Xcode project. This limitation is due to security concerns for hybrid apps that may contain sensitive JavaScript code that could be exploited if visible. Let's check our project's embedded HTML page console. First, open desktop Safari on your Mac and enable developer mode. Launch the Preferences option. Under the Advanced tab, ensure that the Show develop menu in menu bar option is checked, as shown in the following screenshot: Next, let's rerun our Xcode project, start up iOS Simulator and then rerun our page. Once our app is running with the Loop Completed result showing, open desktop Safari and click Develop, then iOS Simulator, followed by index.html. If you look closely, you will see iOS simulator's UIWebView highlighted in blue when you place the mouse over index.html; a visible page is seen as shown in the following screenshot: Once we release the mouse on index.html, we Safari's Web Inspector window appears featuring our hybrid iOS app's DOM and console information. The Safari's Web Inspector is pretty similar to Chrome's Developer tools in terms of feature sets; the panels used in the Developer tools also exist as icons in Web Inspector. Now let's select the Console panel in Web Inspector. Here, we can see our full console window including our Timer console.time function test included in the for loop. As we can see in the following screenshot, the loop took 0.081 milliseconds to process inside iOS. Comparing UIWebView with Mobile Safari What if we wanted to take our code and move it to Mobile Safari to test? This is easy enough; as mentioned earlier in the article, we can drag-and-drop the index.html file into our iOS Simulator, and then the OS will open the mobile version of Safari and load the page for us. With that ready, we will need to reconnect Safari Web Inspector to the iOS Simulator and reload the page. Once that's done, we can see that our console.time function is a bit faster; this time it's roughly 0.07 milliseconds, which is a full .01 milliseconds faster than UIWebView, as shown here: For a small app, this is minimal in terms of a difference in performance. But, as an application gets larger, the delay in these JavaScript processes gets longer and longer. We can also debug the app using the debugging inspector in the Safari's Web Inspector tool. Click Debugger in the top menu panel in Safari's Web Inspector. We can add a break point to our embedded script by clicking a line number and then refreshing the page with Command + R. In the following screenshot, we can see the break occurring on page load, and we can see our scope variable displayed for reference in the right panel: We can also check page load times using the timeline inspector. Click Timelines at the top of the Web Inspector and now we will see a timeline similar to the Resources tab found in Chrome's Developer tools. Let's refresh our page with Command + R on our keyboard; the timeline then processes the page. Notice that after a few seconds, the timeline in the Web Inspector stops when the page fully loads, and all JavaScript processes stop. This is a nice feature when you're working with the Safari Web Inspector as opposed to Chrome's Developer tools. Common ways to improve hybrid performance With hybrid apps, we can use all the techniques for improving performance using a build system such as Grunt.js or Gulp.js with NPM, using JSLint to better optimize our code, writing code in an IDE to create better structure for our apps, and helping to check for any excess code or unused variables in our code. We can use best performance practices such as using strings to apply an HTML page (like the innerHTML property) rather than creating objects for them and applying them to the page that way, and so on. Sadly, the fact that hybrid apps do not perform as well as native apps still holds true. Now, don't let that dismay you as hybrid apps do have a lot of good features! Some of these are as follows: They are (typically) faster to build than using native code They are easier to customize They allow for rapid prototyping concepts for apps They are easier to hand off to other JavaScript developers rather than finding a native developer They are portable; they can be reused for another platform (with some modification) for Android devices, Windows Modern apps, Windows Phone apps, Chrome OS, and even Firefox OS They can interact with native code using helper libraries such as Cordova At some point, however, application performance will be limited to the hardware of the device, and it's recommended you move to native code. But, how do we know when to move? Well, this can be done using Color Blended Layers. The Color Blended Layers option applies an overlay that highlights slow-performing areas on the device display, for example, green for good performance and red for slow performance; the darker the color is, the more impactful will be the performance result. Rerun your app using Xcode and, in the Mac OS toolbar for iOS Simulator, select Debug and then Color Blended Layers. Once we do that, we can see that our iOS Simulator shows a green overlay; this shows us how much memory iOS is using to process our rendered view, both native and non-native code, as shown here: Currently, we can see a mostly green overlay with the exception of the status bar elements, which take up more render memory as they overlay the web view and have to be redrawn over that object repeatedly. Let's make a copy of our project and call it JS_Performance_CBL, and let's update our index.html code with this code sample, as shown in the following screenshot: Here, we have a simple page with an empty div; we also have a button with an onclick function called start. Our start function will update the height continuously using the setInterval function, increasing the height every millisecond. Our empty div also has a background gradient assigned to it with an inline style tag. CSS background gradients are typically a huge performance drain on mobile devices as they can potentially re-render themselves over and over as the DOM updates itself. Some other issues include listener events; some earlier or lower-end devices do not have enough RAM to apply an event listener to a page. Typically, it's a good practice to apply onclick attributes to HTML either inline or through JavaScript. Going back to the gradient example, let's run this in iOS Simulator and enable Color Blended Layers after clicking our HTML button to trigger the JavaScript animation. As expected, our div element that we've expanded now has a red overlay indicating that this is a confirmed performance issue, which is unavoidable. To correct this, we would need to remove the CSS gradient background, and it would show as green again. However, if we had to include a gradient in accordance with a design spec, a native version would be required. When faced with UI issues such as these, it's important to understand tools beyond normal developer tools and Web Inspectors, and take advantage of the mobile platform tools that provide better analysis of our code. Now, before we wrap this article, let's take note of something specific for iOS web views. The WKWebView framework At the time of writing, Apple has announced the WebKit framework, a first-party iOS library intended to replace UIWebView with more advanced and better performing web views; this was done with the intent of replacing apps that rely on HTML5 and JavaScript with better performing apps as a whole. The WebKit framework, also known in developer circles as WKWebView, is a newer web view that can be added to a project. WKWebView is also the base class name for this framework. This framework includes many features that native iOS developers can take advantage of. These include listening for function calls that can trigger native Objective-C or Swift code. For JavaScript developers like us, it includes a faster JavaScript runtime called Nitro, which has been included with Mobile Safari since iOS6. Hybrid apps have always run worse that native code. But with the Nitro JavaScript runtime, HTML5 has equal footing with native apps in terms of performance, assuming that our view doesn't consume too much render memory as shown in our color blended layers example. WKWebView does have limitations though; it can only be used for iOS8 or higher and it doesn't have built-in Storyboard or XIB support like UIWebView. So, using this framework may be an issue if you're new to iOS development. Storyboards are simply XML files coded in a specific way for iOS user interfaces to be rendered, while XIB files are the precursors to Storyboard. XIB files allow for only one view whereas Storyboards allow multiple views and can link between them too. If you are working on an iOS app, I encourage you to reach out to your iOS developer lead and encourage the use of WKWebView in your projects. For more information, check out Apple's documentation of WKWebView at their developer site at https://developer.apple.com/library/IOs/documentation/WebKit/Reference/WKWebView_Ref/index.html. Summary In this article, we learned the basics of creating a hybrid-application for iOS using HTML5 and JavaScript; we learned about connecting the Safari Web Inspector to our HTML page while running an application in iOS Simulator. We also looked at Color Blended Layers for iOS Simulator, and saw how to test for performance from our JavaScript code when it's applied to device-rendering performance issues. Now we are down to the wire. As for all JavaScript web apps before they go live to a production site, we need to smoke-test our JavaScript and web app code and see if we need to perform any final improvements before final deployment. Resources for Article: Further resources on this subject: GUI Components in Qt 5 [article] The architecture of JavaScriptMVC [article] JavaScript Promises – Why Should I Care? [article]
Read more
  • 0
  • 0
  • 9664

article-image-testing-our-application-ios-device
Packt
01 Apr 2015
10 min read
Save for later

Testing our application on an iOS device

Packt
01 Apr 2015
10 min read
In this article by Michelle M. Fernandez, author of the book Corona SDK Mobile Game Development Beginner's Guide, we can upload our first Hello World application on an iOS device, we need to log in into our Apple developer account so that we can create and install our signing certificates on our development machine. If you haven't created a developer account yet, do so by going to http://developer.apple.com/programs/ios/. Remember that there is a fee of $99 a year to become an Apple developer. (For more resources related to this topic, see here.) The Apple developer account is only applied to users developing on Mac OS X. Make sure that your version of Xcode is the same or newer than the version of the OS on your phone. For example, if you have version 5.0 of the iPhone OS installed, you will need Xcode that is bundled with the iOS SDK version 5.0 or later. Time for action – obtaining the iOS developer certificate Make sure that you're signed up for the developer program; you will need to use the Keychain Access tool located in /Applications/Utilities so that you can create a certificate request. A valid certificate must sign all iOS applications before they can be run on an Apple device in order to do any kind of testing. The following steps will show you how to create an iOS developer certificate: Go to Keychain Access | Certificate Assistant | Request a Certificate From a Certificate Authority: In the User Email Address field, type in the e-mail address you used when you registered as an iOS developer. For Common Name, enter your name or team name. Make sure that the name entered matches the information that was submitted when you registered as an iOS developer. The CA Email Address field does not need to be filled in, so you can leave it blank. We are not e-mailing the certificate to a Certificate Authority (CA). Check Saved to disk and Let me specify key pair information. When you click on Continue, you will be asked to choose a save location. Save your file at a destination where you can locate it easily, such as your desktop. In the following window, make sure that 2048 bits is selected for the Key Size and RSA for the Algorithm, and then click on Continue. This will generate the key and save it to the location you specified. Click on Done in the next window. Next, go to the Apple developer website at http://developer.apple.com/, click on iOS Dev Center, and log in to your developer account. Select Certificates, Identifiers & Profiles under iOS Developer Program on the right-hand side of the screen and navigate to Certificates under iOS Apps. Select the + icon on the right-hand side of the page. Under Development, click on the iOS App Development radio button. Click on the Continue button till you reach the screen to generate your certificate: Click on the Choose File button and locate your certificate file that you saved to your desktop, and then, click on the Generate button. Upon hitting Generate, you will get the e-mail notification you specified in the CA request form from Keychain Access, or you can download it directly from the developer portal. The person who created the certificate will get this e-mail and can approve the request by hitting the Approve button. Click on the Download button and save the certificate to a location that is easy to find. Once this is completed, double-click on the file, and the certificate will be added automatically in the Keychain Access. What just happened? We now have a valid certificate for iOS devices. The iOS Development Certificate is used for development purposes only and valid for about a year. The key pair is made up of your public and private keys. The private key is what allows Xcode to sign iOS applications. Private keys are available only to the key pair creator and are stored in the system keychain of the creator's machine. Adding iOS devices You are allowed to assign up to 100 devices for development and testing purposes in the iPhone Developer Program. To register a device, you will need the Unique Device Identification (UDID) number. You can find this in iTunes and Xcode. Xcode To find out your device's UDID, connect your device to your Mac and open Xcode. In Xcode, navigate to the menu bar, select Window, and then click on Organizer. The 40 hex character string in the Identifier field is your device's UDID. Once the Organizer window is open, you should see the name of your device in the Devices list on the left-hand side. Click on it and select the identifier with your mouse, copying it to the clipboard. Usually, when you connect a device to Organizer for the first time, you'll receive a button notification that says Use for Development. Select it and Xcode will do most of the provisioning work for your device in the iOS Provisioning Portal. iTunes With your device connected, open iTunes and click on your device in the device list. Select the Summary tab. Click on the Serial Number label to show the Identifier field and the 40-character UDID. Press Command + C to copy the UDID to your clipboard. Time for action – adding/registering your iOS device To add a device to use for development/testing, perform the following steps: Select Devices in the Developer Portal and click on the + icon to register a new device. Select the Register Device radio button to register one device. Create a name for your device in the Name field and put your device's UDID in the UDID field by pressing Command + V to paste the number you have saved on the clipboard. Click on Continue when you are done and click on Register once you have verified the device information. Time for action – creating an App ID Now that you have added a device to the portal, you will need to create an App ID. An App ID has a unique 10-character Apple ID Prefix generated by Apple and an Apple ID Suffix that is created by the Team Admin in the Provisioning Portal. An App ID could looks like this: 7R456G1254.com.companyname.YourApplication. To create a new App ID, use these steps: Click on App IDs in the Identifiers section of the portal and select the + icon. Fill out the App ID Description field with the name of your application. You are already assigned an Apple ID Prefix (also known as a Team ID). In the App ID Suffix field, specify a unique identifier for your app. It is up to you how you want to identify your app, but it is recommended that you use the reverse-domain style string, that is, com.domainname.appname. Click on Continue and then on Submit to create your App ID. You can create a wildcard character in the bundle identifier that you can share among a suite of applications using the same Keychain access. To do this, simply create a single App ID with an asterisk (*) at the end. You would place this in the field for the bundle identifier either by itself or at the end of your string, for example, com.domainname.*. More information on this topic can be found in the App IDs section of the iOS Provisioning Portal at https://developer.apple.com/ios/manage/bundles/howto.action. What just happened? All UDIDs are unique on every device, and we can locate them in Xcode and iTunes. When we added a device in the iOS Provisioning Portal, we took the UDID, which consists of 40 hex characters, and made sure we created a device name so that we could identify what we're using for development. We now have an App ID for the applications we want to install on a device. An App ID is a unique identifier that iOS uses to allow your application to connect to the Apple Push Notification service, share keychain data between applications, and communicate with external hardware accessories you wish to pair your iOS application with. Provisioning profiles A provisioning profile is a collection of digital entities that uniquely ties apps and devices to an authorized iOS Development Team and enables a device to be used to test a particular app. Provisioning profiles define the relationship between apps, devices, and development teams. They need to be defined for both the development and distribution aspects of an app. Time for action – creating a provisioning profile To create a provisioning profile, go to the Provisioning Profiles section of the Developer Portal and click on the + icon. Perform the following steps: Select the iOS App Development radio button under the Development section and then select Continue. Select the App ID you created for your application in the pull-down menu and click on Continue. Select the certificate you wish to include in the provisioning profile and then click on Continue. Select the devices you wish to authorize for this profile and click on Continue. Create a Profile Name and click on the Generate button when you are done: Click on the Download button. While the file is downloading, launch Xcode if it's not already open and press Shift + Command + 2 on the keyboard to open Organizer. Under Library, select the Provisioning Profiles section. Drag your downloaded .mobileprovision file to the Organizer window. This will automatically copy your .mobileprovision file to the proper directory. What just happened? Devices that have permission within the provisioning profile can be used for testing as long as the certificates are included in the profile. One device can have multiple provisioning profiles installed. Application icon Currently, our app has no icon image to display on the device. By default, if there is no icon image set for the application, you will see a light gray box displayed along with your application name below it once the build has been loaded to your device. So, launch your preferred creative developmental tool and let's create a simple image. The application icon for standard resolution iPad2 or iPad mini image file is 76 x 76 px PNG. The image should always be saved as Icon.png and must be located in your current project folder. iPhone/iPod touch devices that support retina display need an additional high resolution 120 x 120 px and iPad or iPad mini have an icon of 152 x 152 px named as Icon@2x.png. The contents of your current project folder should look like this: Hello World/       name of your project folderIcon.png           required for iPhone/iPod/iPadIcon@2x.png   required for iPhone/iPod with Retina displaymain.lua In order to distribute your app, the App Store requires a 1024 x 1024 pixel version of the icon. It is best to create your icon at a higher resolution first. Refer to the Apple iOS Human Interface Guidelines for the latest official App Store requirements at http://developer.apple.com/library/ios/#documentation/userexperience/conceptual/mobilehig/Introduction/Introduction.html. Creating an application icon is a visual representation of your application name. You will be able to view the icon on your device once you compile a build together. The icon is also the image that launches your application. Summary In this article, we covered how to test your app on an iOS device and register your iOS device. Resources for Article: Further resources on this subject: Linking OpenCV to an iOS project [article] Creating a New iOS Social Project [article] Sparrow iOS Game Framework - The Basics of Our Game [article]
Read more
  • 0
  • 0
  • 11080
article-image-creating-games-cocos2d-x-easy-and-100-percent-free-0
Packt
01 Apr 2015
5 min read
Save for later

Creating Games with Cocos2d-x is Easy and 100 percent Free

Packt
01 Apr 2015
5 min read
In this article by Raydelto Hernandez, the author of the book Building Android games with Cocos2d-x, we will talk about the Cocos2d-x game engine, which is widely used to create Android games. The launch of the Apple App Store back in 2008 leveraged the reach capacity of indie game developers who since its occurrence are able to reach millions of users and compete with large companies, outperforming them in some situations. This reality led the trend of creating reusable game engines, such as Cocos2d-iPhone, which is written natively using Objective-C by the Argentine iPhone developer, Ricardo Quesada. Cocos2d-iPhone allowed many independent developers to reach the top charts of downloads. (For more resources related to this topic, see here.) Picking an existing game engine is a smart choice for indies and large companies since it allows them to focus on the game logic rather than rewriting core features over and over again. Thus, there are many game engines out there with all kinds of licenses and characteristics. The most popular game engines for mobile systems right now are Unity, Marmalade, and Cocos2d-x; the three of them have the capabilities to create 2D and 3D games. Determining which one is the best in terms of ease of use and availability of tools may be debatable, but there is one objective fact, which we can mention that could be easily verified. Among these three engines, Cocos2d-x is the only one that you can use for free no matter how much money you make using it. We highlighted in this article's title that Cocos2d-x is completely free. This was emphasized because the other two frameworks also allow some free usage; nevertheless, both of these at some point require a payment for the usage license. In order to understand why Cocos2d-x is still free and open source, we need to understand how this tool was born. Ricardo, an enthusiastic Python programmer, often participated in game creation challenges that required participants to develop games from scratch within a week. Back in those days, Ricardo and his team rewrote the core engine for each game until they came up with the idea of creating a framework to encapsulate core game capabilities. These capabilities could be used on any two-dimensional game to make it open source, so contributions could be received worldwide. This is why Cocos2d was originally written for fun. With the launch of the first iPhone in 2007, Ricardo led the development of the port of the Cocos2d Python framework to the iPhone platform using its native language, Objective-C. Cocos2d-iPhone quickly became popular among indie game developers, some of them turning into Appillionaires, as Chris Stevens called these individuals and enterprises that made millions of dollars during the App Store bubble period. This phenomenon made game development companies look at this framework created by hobbyists as a tool to create their products. Zynga was one of the first big companies to adopt Cocos2d as their framework to deliver their famous Farmville game to iPhone in 2009. This company has been trading on NASDAQ since 2011 and has more than 2,000 employees. In July 2010, a C++ port of the Cocos2d iPhone called Cocos2d-x, was written in China with the objective of taking the power of this framework to other platforms, such as the Android operating system, which by that time was gaining market share at a spectacular rate. In 2011, this Cocos2d port was acquired by Chukong Technologies, the third largest mobile game development company in China, who later hired the original Cocos2d-IPhone author to join their team. Today, Cocos2d-x-based games dominate the top grossing charts of Google Play and the App Store, especially in Asia. Recognized companies and leading studios, such as Konami, Zynga, Bandai Namco, Wooga, Disney Mobile, and Square Enix are using Cocos2d-x in their games. Currently, there are 400,000 developers working on adding new functionalities and making this framework as stable as possible. These include engineers from Google, ARM, Intel, BlackBerry, and Microsoft who officially support the ports of their products, such as Windows Phone, Windows, Windows Metro Interface, and they're planning to support Cocos2d-x for the Xbox in this year. Cocos2d-x is a very straightforward engine that requires a little learning to grasp it. I teach game development courses at many universities using this framework; during the first week, the students are capable of creating a game with the complexity of the famous title Doodle Jump. This can be easily achieved because the framework provides us all the single components that are required for our game, such as physics, audio handling, collision detection, animation, networking, data storage, user input, map rendering, scene transitions, 3D rendering, particle systems rendering, font handling, menu creation, displaying forms, threads handling, and so on. This abstracts us from the low-level logic and allows us to focus on the game logic. Summary In conclusion, if you are willing to learn how to develop games for mobile platforms, I strongly recommend you to learn and use the Cocos2d-x framework because it is easy to use, is totally free, is an open source. This means that you can better understand it by reading its source, you could modify it if needed, and you have the warranty that you will never be forced to pay a license fee if your game becomes a hit. Another big advantage of this framework is its highly available documentation, including the Packt Publishing collection of Cocos2d-x game development books. Resources for Article: Further resources on this subject: Moving the Space Pod Using Touch [article] Why should I make cross-platform games? [article] Animations in Cocos2d-x [article]
Read more
  • 0
  • 0
  • 5502

article-image-introduction-testing-angularjs-directives
Packt
01 Apr 2015
14 min read
Save for later

An introduction to testing AngularJS directives

Packt
01 Apr 2015
14 min read
In this article by Simon Bailey, the author of AngularJS Testing Cookbook, we will cover the following recipes: Starting with testing directives Setting up templateUrl Searching elements using selectors Accessing basic HTML content Accessing repeater content (For more resources related to this topic, see here.) Directives are the cornerstone of AngularJS and can range in complexity providing the foundation to many aspects of an application. Therefore, directives require comprehensive tests to ensure they are interacting with the DOM as intended. This article will guide you through some of the rudimentary steps required to embark on your journey to test directives. The focal point of many of the recipes revolves around targeting specific HTML elements and how they respond to interaction. You will learn how to test changes on scope based on a range of influences and finally begin addressing testing directives using Protractor. Starting with testing directives Testing a directive involves three key steps that we will address in this recipe to serve as a foundation for the duration of this article: Create an element. Compile the element and link to a scope object. Simulate the scope life cycle. Getting ready For this recipe, you simply need a directive that applies a scope value to the element in the DOM. For example: angular.module('chapter5', []) .directive('writers', function() {    return {      restrict: 'E',      link: function(scope, element) {        element.text('Graffiti artist: ' + scope.artist);      }    }; }); How to do it… First, create three variables accessible across all tests:     One for the element: var element;     One for scope: var scope;     One for some dummy data to assign to a scope value: var artist = 'Amara Por Dios'; Next, ensure that you load your module: beforeEach(module('chapter5')); Create a beforeEach function to inject the necessary dependencies and create a new scope instance and assign the artist to a scope: beforeEach(inject(function ($rootScope, $compile) { scope = $rootScope.$new(); scope.artist = artist; })); Next, within the beforeEach function, add the following code to create an Angular element providing the directive HTML string: element = angular.element('<writers></writers>'); Compile the element providing our scope object: $compile(element)(scope); Now, call $digest on scope to simulate the scope life cycle: scope.$digest(); Finally, to confirm whether these steps work as expected, write a simple test that uses the text() method available on the Angular element. The text() method will return the text contents of the element, which we then match against our artist value: it('should display correct text in the DOM', function() { expect(element.text()).toBe('Graffiti artist: ' + artist); }); Here is what your code should look like to run the final test: var scope; var element; var artist;   beforeEach(module('chapter5'));   beforeEach(function() { artist = 'Amara Por Dios'; });   beforeEach(inject(function($compile) { element = angular.element('<writers></writers>'); scope.artist = artist; $compile(element)(scope); scope.$digest(); }));   it('should display correct text in the DOM', function() {    expect(element.text()).toBe('Graffiti artist: ' + artist); }); How it works… In step 4, the directive HTML tag is provided as a string to the angular.element function. The angular element function wraps a raw DOM element or an HTML string as a jQuery element if jQuery is available; otherwise, it defaults to using Angular's jQuery lite which is a subset of jQuery. This wrapper exposes a range of useful jQuery methods to interact with the element and its content (for a full list of methods available, visit https://docs.angularjs.org/api/ng/function/angular.element). In step 6, the element is compiled into a template using the $compile service. The $compile service can compile HTML strings into a template and produces a template function. This function can then be used to link the scope and the template together. Step 6 demonstrates just this, linking the scope object created in step 3. The final step to getting our directive in a testable state is in step 7 where we call $digest to simulate the scope life cycle. This is usually part of the AngularJS life cycle within the browser and therefore needs to be explicitly called in a test-based environment such as this, as opposed to end-to-end tests using Protractor. There's more… One beforeEach() method containing the logic covered in this recipe can be used as a reference to work from for the rest of this article: beforeEach(inject(function($rootScope, $compile) { // Create scope scope = $rootScope.$new(); // Replace with the appropriate HTML string element = angular.element('<deejay></deejay>'); // Replace with test scope data scope.deejay = deejay; // Compile $compile(element)(scope); // Digest scope.$digest(); })); See also The Setting up templateUrl recipe The Searching elements using selectors recipe The Accessing basic HTML content recipe The Accessing repeater content recipe Setting up templateUrl It's fairly common to separate the template content into an HTML file that can then be requested on demand when the directive is invoked using the templateUrl property. However, when testing directives that make use of the templateUrl property, we need to load and preprocess the HTML files to AngularJS templates. Luckily, the AngularJS team preempted our dilemma and provided a solution using Karma and the karma-ng-html2js-preprocessor plugin. This recipe will show you how to use Karma to enable us to test a directive that uses the templateUrl property. Getting ready For this recipe, you will need to ensure the following: You have installed Karma You installed the karma-ng-html2js-preprocessor plugin by following the instructions at https://github.com/karma-runner/karma-ng-html2js-preprocessor/blob/master/README.md#installation. You configured the karma-ng-html2js-preprocessor plugin by following the instructions at https://github.com/karma-runner/karma-ng-html2js-preprocessor/blob/master/README.md#configuration. Finally, you'll need a directive that loads an HTML file using templateUrl and for this example, we apply a scope value to the element in the DOM. Consider the following example: angular.module('chapter5', []) .directive('emcees', function() {    return {      restrict: 'E',      templateUrl: 'template.html',      link: function(scope, element) {        scope.emcee = scope.emcees[0];      }    }; }) An example template could be as simple as what we will use for this example (template.html): <h1>{{emcee}}</h1> How to do it… First, create three variables accessible across all tests:     One for the element: var element;     One for the scope: var scope;     One for some dummy data to assign to a scope value: var emcees = ['Roxanne Shante', 'Mc Lyte']; Next, ensure that you load your module: beforeEach(module('chapter5')); We also need to load the actual template. We can do this by simply appending the filename to the beforeEach function we just created in step 2: beforeEach(module('chapter5', 'template.html')); Next, create a beforeEach function to inject the necessary dependencies and create a new scope instance and assign the artist to a scope: beforeEach(inject(function ($rootScope, $compile) { scope = $rootScope.$new(); Scope.emcees = emcees; })); Within the beforeEach function, add the following code to create an Angular element providing the directive HTML string:    element = angular.element('<emcees></emcees>'); Compile the element providing our scope object: $compile(element)(scope); Call $digest on scope to simulate the scope life cycle: scope.$digest(); Next, create a basic test to establish that the text contained within the h1 tag is what we expect: it('should set the scope property id to the correct initial value', function () {}); Now, retrieve a reference to the h1 tag using the find() method on the element providing the tag name as the selector: var h1 = element.find('h1'); Finally, add the expectation that the h1 tag text matches our first emcee from the array we provided in step 4: expect(h1.text()).toBe(emcees[0]); You will see the following passing test within your console window: How it works… The karma-ng-html2js-preprocessor plugin works by converting HTML files into JS strings and generates AngularJS modules that we load in step 3. Once loaded, AngularJS makes these modules available by putting the HTML files into the $templateCache. There are libraries available to help incorporate this into your project build process, for example using Grunt or Gulp. There is a popular example specifically for Gulp at https://github.com/miickel/gulp-angular-templatecache. Now that the template is available, we can access the HTML content using the compiled element we created in step 5. In this recipe, we access the text content of the element using the find() method. Be aware that if using the smaller jQuery lite subset of jQuery, there are certain limitations compared to the full-blown jQuery version. The find() method in particular is limited to look up by tag name only. To read more about the find() method, visit the jQuery API documentation at http://api.jquery.com/find. See also The Starting with testing directives recipe Searching elements using selectors Directives, as you should know, attach special behavior to a DOM element. When AngularJS compiles and returns the element on which the directive is applied, it is wrapped by either jqLite or jQuery. This exposes an API on the element, offering many useful methods to query the element and its contents. In this recipe, you will learn how to use these methods to retrieve elements using selectors. Getting ready Follow the logic to define a beforeEach() function with the relevant logic to set up a directive as outlined in the Starting with testing directives recipe in this article. For this recipe, you can replicate the template that I suggested in the first recipe's There's more… section. For the purpose of this recipe, I tested against a property on scope named deejay: var deejay = { name: 'Shortee', style: 'turntablism' }; You can replace this with whatever code you have within the directive you're testing. How to do it… First, create a basic test to establish that the HTML code contained within an h2 tag is as we expected: it('should return an element using find()', function () {}); Next, retrieve a reference to the h2 tag using the find() method on the element providing the tag name as the selector: var h2 = element.find('h2'); Finally, we create an expectation that the element is actually defined: expect(h2[0]).toBeDefined(); How it works… In step 2, we use the find() method with the h2 selector to test against in step 3's expectation. Remember, the element returned is wrapped by jqLite or jQuery. Therefore, even if the element is not found, the object returned will have jQuery-specific properties; this means that we cannot run an expectation on the element alone being defined. A simple way to determine if the element itself is indeed defined is to access it via jQuery's internal array of DOM objects, typically the first. So, this is why in our recipe we run an expectation against element[0] as opposed to element itself. There's more… Here is an example using the querySelector() method. The querySelector() method is available on the actual DOM so we need to access it on an actual HTML element and not the jQuery wrapped element. The following code shows the selector we use in a CSS selector: it('should return an element using querySelector and css selector', function() { var elementByClass = element[0].querySelector('.deejay- style'); expect(elementByClass).toBeDefined(); }); Here is a another example using the querySelector() method that uses an id selector: it(should return an element using querySelector and id selector', function() { var elementByClass = element[0].querySelector(' #deejay_name'); expect(elementByClass).toBeDefined(); }); You can read more about the querySelector() method at https://developer.mozilla.org/en-US/docs/Web/API/document.querySelector. See also The Starting with testing directives recipe The Accessing basic HTML content recipe Accessing basic HTML content A substantial number of directive tests will involve interacting with the HTML content within the rendered HTML template. This recipe will teach you how to test whether a directive's HTML content is as expected. Getting ready Follow the logic to define a beforeEach() function with the relevant logic to set up a directive as outlined in the Starting with testing directives recipe in this article. For this recipe, you can replicate the template that I suggested in the first recipe's There's more… section. For the purpose of this recipe, I will test against a property on a scope named deejay: var deejay = { name: 'Shortee', style: 'turntablism' }; You can replace this with whatever code you have within the directive you're testing. How to do it… First, create a basic test to establish that the HTML code contained within a h2 tag is as we expected: it('should display correct deejay data in the DOM', function () {}); Next, retrieve a reference to the h2 tag using the find() method on the element providing the tag name as the selector: var h2 = element.find('h2'); Finally, using the html() method on the returned element from step 2, we can get the HTML contents within an expectation that the h2 tag HTML code matches our scope's deejay name: expect(h2.html()).toBe(deejay.name); How it works… We made heavy use of the jQuery (or jqLite) library methods available for our element. In step 2, we use the find() method with the h2 selector. This returns a match for us to further utilize in step 3, in our expectation where we access the HTML contents of the element using the html() method this time (http://api.jquery.com/html/). There's more… We could also run a similar expectation for text within our h2 element using the text() method (http://api.jquery.com/text/) on the element, for example: it('should retrieve text from <h2>', function() { var h2 = element.find('h2'); expect(h2.text()).toBe(deejay.name); }); See also The Starting with testing directives recipe The Searching elements using selectors recipe Accessing repeater content AngularJS facilitates generating repeated content with ease using the ngRepeat directive. In this recipe, we'll learn how to access and test repeated content. Getting ready Follow the logic to define a beforeEach() function with the relevant logic to set up a directive as outlined in the Starting with testing directives recipe in this article. For this recipe, you can replicate the template that I suggested in the first recipe's There's more… section. For the purpose of this recipe, I tested against a property on scope named breakers: var breakers = [{ name: 'China Doll' }, { name: 'Crazy Legs' }, { name: 'Frosty Freeze' }]; You can replace this with whatever code you have within the directive you're testing. How to do it… First, create a basic test to establish that the HTML code contained within the h2 tag is as we expected: it('should display the correct breaker name', function () {}); Next, retrieve a reference to the li tag using the find() method on the element providing the tag name as the selector: var list = element.find('li'); Finally, targeting the first element in the list, we retrieve the text content expecting it to match the first item in the breakers array: expect(list.eq(0).text()).toBe('China Doll'); How it works… In step 2, the find() method using li as the selector will return all the list items. In step 3, using the eq() method (http://api.jquery.com/eq/) on the returned element from step 2, we can get the HTML contents at a specific index, zero in this particular case. As the returned object from the eq() method is a jQuery object, we can call the text() method, which immediately after that will return the text content of the element. We can then run an expectation that the first li tag text matches the first breaker within the scope array. See also The Starting with testing directives recipe The Searching elements using selectors recipe The Accessing basic HTML content recipe Summary In this article you have learned to focus on testing changes within a directive based on interaction from either UI events or application updates to the model. Directives are one of the important jewels of AngularJS and can range in complexity. They can provide the foundation to many aspects of the application and therefore require comprehensive tests. Resources for Article: Further resources on this subject: The First Step [article] AngularJS Performance [article] Our App and Tool Stack [article]
Read more
  • 0
  • 0
  • 2076
Modal Close icon
Modal Close icon