Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-preparing-build-your-own-gis-application
Packt
15 Jun 2015
12 min read
Save for later

Preparing to Build Your Own GIS Application

Packt
15 Jun 2015
12 min read
In this article by Karim Bahgat, author of the book Python Geospatial Development Essentials, we will see how to design your own Geographic Information Systems (GIS) application. You want to create a desktop application, in other words, a user interface, that helps you or others create, process, analyze, and visualize geographic data. This article will be your step-by-step guide toward that goal. (For more resources related to this topic, see here.) We assume that you are someone who enjoys programming and being creative but are not necessarily a computer science guru, Python expert, or seasoned GIS analyst. To successfully proceed with this article, it is recommended that you have a basic introductory knowledge of Python programming that includes classes, methods, and the Tkinter toolkit, as well as some core GIS concepts. If you are a newcomer to some of these, we will still cover some of the basics, but you will need to have the interest and ability to follow along at a fast pace. In this introductory article, you will cover the following: Learn some of the benefits of creating a GIS application from scratch Set up your computer, so you can follow the article instructions Become familiar with the roadmap toward creating our application Why reinvent the wheel? The first step in preparing ourselves for this article is in convincing ourselves why we want to make our own GIS application, as well as to be clear about our motives. Spatial analysis and GIS have been popular for decades and there is plenty of GIS software out there, so why go through the trouble of reinventing the wheel? Firstly, we aren't really reinventing the wheel, since Python can be extended with plenty of third-party libraries that take care of most of our geospatial needs. For me, the main motivation stems from the problem that most of today's GIS applications are aimed at highly capable and technical users who are well-versed in GIS or computer science, packed with a dizzying array of buttons and options that will scare off many an analyst. We believe that there is a virtue in trying to create a simpler and more user-friendly software for beginner GIS users or even the broader public, without having to start completely from scratch. This way, we also add more alternatives for users to choose from, as supplements to the current GIS market dominated by a few major giants, notably ArcGIS and QGIS, but also others such as GRASS, uDig, gvSIG, and more. Another particularly exciting reason to create your own GIS from scratch is to make your own domain-specific special purpose software for any task you can imagine, whether it is a water flow model GIS, an ecological migrations GIS, or even a GIS for kids. Such specialized tasks that would usually require many arduous steps in an ordinary GIS, could be greatly simplified into a single button and accompanied with suitable functionality, design layout, icons, and colors. One such example is the Crime Analytics for Space-Time (CAST) software produced by the GeoDa Center at Arizona State University, seen in the following picture: Also, by creating your GIS from scratch, it is possible to have greater control of the size and portability of your application. This can enable you to go small—letting your application have faster startup time, and travel the Internet or on a USB-stick easily. Although storage space itself is not as much of an issue these days, from a user's perspective, installing a 200 MB application is still a greater psychological investment with a greater toll in terms of willingness to try it than a mere 30 MB application (all else being equal). This is particularly true in the realm of smartphones and tablets, a very exciting market for special-purpose geospatial apps. While the specific application we make in this article will not be able to run on iOS or Android devices, it will run on Windows 8-based hybrid tablets, and can be rebuilt around a different GUI toolkit in order to support iOS. Finally, the utility and philosophy of free and open source software may be an important motivation for some of you. Many people today, learn to appreciate open source GIS after losing access to subscription-based applications like ArcGIS when they complete their university education or change their workplace. By developing your own open source GIS application and sharing with others, you can contribute back to and become part of the community that once helped you. Setting up your computer In this article, we follow steps on how to make an application that is developed in a Windows environment. This does not mean that the application cannot be developed on Mac OS X or Linux, but those platforms may have slightly different installation instructions and may require compiling of the binary code that is outside the scope of this article. Therefore, we leave that choice up to the reader. In this article, which focuses on Windows, we avoid the problem of compiling it altogether, using precompiled versions where possible (more on this later). The development process itself will be done using Python 2.7, specifically the 32-bit version, though 64-bit can theoretically be used as well (note that this is the bit version of your Python installation and has nothing to do with the bit version of your operating system). Although there exists many newer versions, version 2.7 is the most widely supported in terms of being able to use third-party packages. It has also been reported that the version 2.7 will continue to be actively developed and promoted until the year 2020. It will still be possible to use after support has ended. If you do not already have version 2.7, install it now, by following these steps: Go to https://www.python.org/. Under Downloads click on download the latest 32-bit version of Python 2.7 for Windows, which at the time of this writing is Python 2.7.9. Download and run the installation program. For the actual code writing and editing, we will be using the built-in Python Interactive Development Environment (IDLE), but you may of course use any code editor you want. The IDLE lets you write long scripts that can be saved to files and offers an interactive shell window to execute one line at a time. There should be a desktop or start-menu link to Python IDLE after installing Python. Installing third-party packages In order to make our application, we will have to rely on the rich and varied ecosystem of third-party packages that already exists for GIS usage. The Python Package Index (PyPI) website currently lists more than 240 packages tagged Topic :: Scientific/Engineering :: GIS. For a less overwhelming overview of the more popular GIS-related Python libraries, check out the catalogue at the Python-GIS-Resources website created by the author: http://pythongisresources.wordpress.com/ We will have to define which packages to use and install, and this depends on the type of application we are making. What we want to make in this article is a lightweight, highly portable, extendable, and general-purpose GIS application. For these reasons, we avoid heavy packages like GDAL, NumPy, Matplotlib, SciPy, and Mapnik (weighing in at about 30 MB each or about 150-200 MB if we combine them all together). Instead, we focus on lighter third-party packages specialized for each specific functionality. Dropping these heavy packages is a bold decision, as they contain a lot of functionality, and are reliable, efficient, and a dependency for many other packages. If you decide that you want to use them in an application where size is not an issue, you may want to begin now by installing the multipurpose NumPy and possibly SciPy, both of which have easy-to-use installers from their official websites. The typical way to install Python packages is using pip (included with Python 2.7), which downloads and installs packages directly from the Python Package Index website. Pip is used in the following way: Step 1—open your operating system's command line (not the Python IDLE). On Windows, this is done by searching your system for cmd.exe and running it. Step 2—in the black screen window that pops up, one simply types pip install packagename. This will only work if pip is on your system's environment path. If this is not the case, a quick fix is to simply type the full path to the pip script C:Python27Scriptspip instead of just pip. For C or C++ based packages, it is becoming increasingly popular to make them available as precompiled wheel files ending in .whl, which has caused some confusion on how to install them. Luckily, we can use pip to install these wheel files as well, by simply downloading the wheel and pointing pip to its file path. Since some of our dependencies have multiple purposes and are not unique, we will install these ones now. One of them is the Python Imaging Library (PIL), which we will use for the raster data model and for visualization. Let's go ahead and install PIL for Windows now: Go to https://pypi.python.org/pypi/Pillow/2.6.1. Click on the latest .exe file link for our 32-bit Python 2.7 environment to download the PIL installer, which is currently Pillow-2.6.1.win32-py2.7.exe. Run the installation file. Open the IDLE interactive shell and type import PIL to make sure it was installed correctly. Another central package we will be using is Shapely, used for location testing and geometric manipulation. To install it on Windows, perform the following steps: Go to http://www.lfd.uci.edu/~gohlke/pythonlibs/#shapely. Download the Shapely wheel file that fits our system, looking something like Shapely‑1.5.7‑cp27‑none‑win32.whl. As described earlier, open a command line window and type C:Python27Scriptspip install pathtoShapely‑1.5.7‑cp27‑none‑win32.whl to unpack the precompiled binaries. To make sure it was installed correctly, open the IDLE interactive shell and type import shapely. Imagining the roadmap ahead Before we begin developing our application, it is important that we create a vision of how we want to structure our application. In Python terms, we will be creating a multilevel package with various subpackages and submodules to take care of different parts of our functionality, independently of any user interface. Only on top of this underlying functionality do we create the visual user interface as a way to access and run that underlying code. This way, we build a solid system, and allow power-users to access all the same functionality via Python scripting for greater automation and efficiency, as exists for ArcGIS and QGIS. To setup the main Python package behind our application, create a new folder called pythongis anywhere on your computer. For Python to be able to interpret the folder pythongis as an importable package, it needs to find a file named __init__.py in that folder. Perform the following steps: Open Python IDLE from the Windows start menu. The first window to pop up is the interactive shell. To open the script editing window click on File and New. Click on File and then Save As. In the dialog window that pops up, browse into the pythongis folder, type __init__.py as the filename, and click on Save. There are two main types of GIS data: vector (coordinate-based geometries such as points, lines, and polygons) and raster (a regularly spaced out grid of data points or cells, similar to an image and its pixels). For a more detailed introduction to the differences between vector and raster data, and other basic GIS concepts, we refer the reader to the book Learning Geospatial Analysis with Python, by Joel Lawhead. You can find this book at: https://www.packtpub.com/application-development/learning-geospatial-analysis-python Since vector and raster data are so fundamentally different in all regards, we split our package in two, one for vector and one for raster. Using the same method as earlier, we create two new subpackage folders within the pythongis package; one called vector and one called raster (each with the same aforementioned empty __init__.py file). Thus, the structure of our package will look as follows (note that : package is not part of the folder name): To make our new vector and raster subpackages importable by our top level pythongis package, we need to add the following relative import statements in pythongis/__init__.py: from . import vectorfrom . import raster Throughout the course of this article, we will build the functionality of these two data types as a set of Python modules in their respective folders. Eventually, we want to end up with a GIS application that has only the most basic of geospatial tools so that we will be able to load, save, manage, visualize, and overlay data. As far as our final product goes, since we focus on clarity and simplicity, we do not put too much effort into making it fast or memory efficient. This comes from an often repeated saying among programmers, an example of which is found in Structured Programming with go to Statements, ACM, Computing Surveys 6 (4): premature optimization is the root of all evil – Donald E. Knuth This leaves us with software that works best with small files, which in most cases is good enough. Once you have a working application and you feel that you need support for larger or faster files, then it's up to you if you want to put in the extra effort of optimization. The GIS application you end up with at the end of the article is simple but functional, and is meant to serve as a framework that you can easily build on. To leave you with some ideas to pick up on, we placed various information boxes throughout the article with ways that you can optimize or extend your application. Summary In this article, you learned about why you want to create a GIS application using Python, set up our programming environment, installed some recurring packages, and created your application structure and framework. Resources for Article: Further resources on this subject: Python functions – Avoid repeating code [article] Python 3: Designing a Tasklist Application [article] Geolocating photos on the map [article]
Read more
  • 0
  • 0
  • 8411

article-image-visualizing-my-social-graph-d3js
Packt
24 Oct 2013
7 min read
Save for later

Visualizing my Social Graph with d3.js

Packt
24 Oct 2013
7 min read
(For more resources related to this topic, see here.) The Social Networks Analysis Social Networks Analysis (SNA) is not new, sociologists have been using it for a long time to study human relationships (sociometry), to find communities and to simulate how information or a disease is spread in a population. With the rise of social networking sites such as Facebook, Twitter, LinkedIn, and so on. The acquisition of large amounts of social network data is easier. We can use SNA to get insight about customer behavior or unknown communities. It is important to say that this is not a trivial task and we will come across sparse data and a lot of noise (meaningless data). We need to understand how to distinguish between false correlation and causation. A good start is by knowing our graph through visualization and statistical analysis. Social networking sites bring us the opportunities to ask questions that otherwise are too hard to approach, because polling enough people is time-consuming and expensive. In this article, we will obtain our social network's graph from Facebook (FB) website in order to visualize the relationships between our friends. Finally we will create an interactive visualization of our graph using D3.js. Getting ready The easiest method to get our friends list is by using a third-party application. Netvizz is a Facebook app developed by Bernhard Rieder, which allows exporting social graph data to gdf and tab formats. Netvizz may export information about our friends such as gender, age, locale, posts, and likes. In order to get our social graph from Netvizz we need to access the link below and giving access to your Facebook profile. https://apps.facebook.com/netvizz/ As is shown in the following screenshot, we will create a gdf file from our personal friend network by clicking on the link named here in the Step 2. Then we will download the GDF (Graph Modeling Language) file. Netvizz will give us the number of nodes and edges (links); finally we will click on the gdf file link, as we can see in the following screenshot: The output file myFacebookNet.gdf will look like this: nodedef>name VARCHAR,label VARCHAR,gender VARCHAR,locale VARCHAR,agerankINT23917067,Jorge,male,en_US,10623931909,Haruna,female,en_US,10535702006,Joseph,male,en_US,104503839109,Damian,male,en_US,103532735006,Isaac,male,es_LA,102. . .edgedef>node1 VARCHAR,node2 VARCHAR23917067,3570200623917067,62939583723917067,74734348223917067,75560507523917067,1186286815. . . In the following screenshot we may see the visualization of the graph (106 nodes and 279 links). The nodes represent my friends and the links represent how my friends are connected between them. Transforming GDF to JSON In order to work with the graph in the web with d3.js, we need to transform our gdf file to json format. Firstly, we need to import the libraries numpy and json. import numpy as npimport json The numpy function, genfromtxt, will obtain only the ID and name from the nodes.csv file using the usecols attribute in the 'object' format. nodes = np.genfromtxt("nodes.csv",dtype='object',delimiter=',',skip_header=1,usecols=(0,1)) Then, the numpy function, genfromtxt, will obtain links with the source node and target node from the links.csv file using the usecols attribute in the 'object' format. links = np.genfromtxt("links.csv",dtype='object',delimiter=',',skip_header=1,usecols=(0,1)) The JSON format used in the D3.js Force Layout graph implemented in this article requires transforming the ID (for example, 100001448673085) into a numerical position in the list of nodes. Then, we need to look for each appearance of the ID in the links and replace them by their position in the list of nodes. for n in range(len(nodes)):for ls in range(len(links)):if nodes[n][0] == links[ls][0]:links[ls][0] = nif nodes[n][0] == links[ls][1]:links[ls][1] = n Now, we need to create a dictionary "data" to store the JSON file. data ={} Next, we need to create a list of nodes with the names of the friends in the format as follows: "nodes": [{"name": "X"},{"name": "Y"},. . .] and add it to thedata dictionary.lst = []for x in nodes:d = {}d["name"] = str(x[1]).replace("b'","").replace("'","")lst.append(d)data["nodes"] = lst Now, we need to create a list of links with the source and target in the format as follows: "links": [{"source": 0, "target": 2},{"source": 1, "target":2},. . .] and add it to the data dictionary.lnks = []for ls in links:d = {}d["source"] = ls[0]d["target"] = ls[1]lnks.append(d)data["links"] = lnks Finally, we need to create the file, newJson.json, and write the data dictionary in the file with the function dumps of the json library. with open("newJson.json","w") as f:f.write(json.dumps(data)) The file newJson.json will look as follows: {"nodes": [{"name": "Jorge"},{"name": "Haruna"},{"name": "Joseph"},{"name": "Damian"},{"name": "Isaac"},. . .],"links": [{"source": 0, "target": 2},{"source": 0, "target": 12},{"source": 0, "target": 20},{"source": 0, "target": 23},{"source": 0, "target": 31},. . .]} Graph visualization with D3.js D3.js provides us with the d3.layout.force() function that use the Force Atlas layout algorithm and help us to visualize our graph. First, we need to define the CSS style for the nodes, links, and node labels. <style>.link {fill: none;stroke: #666;stroke-width: 1.5px;}.node circle{fill: steelblue;stroke: #fff;stroke-width: 1.5px;}.node text{pointer-events: none;font: 10px sans-serif;}</style> Then, we need to refer the d3js library. <script src = "http://d3js.org/d3.v3.min.js"></script> Then, we need to define the width and height parameters for the svg container and include into the body tag. var width = 1100,height = 800var svg = d3.select("body").append("svg").attr("width", width).attr("height", height); Now, we define the properties of the force layout such as gravity, distance, and size. var force = d3.layout.force().gravity(.05).distance(150).charge(-100).size([width, height]); Then, we need to acquire the data of the graph using the JSON format. We will configure the parameters for nodes and links. d3.json("newJson.json", function(error, json) {force.nodes(json.nodes).links(json.links).start(); For a complete reference about the d3js Force Layout implementation, visit the link https://github.com/mbostock/d3/wiki/Force-Layout. Then, we define the links as lines from the json data. var link = svg.selectAll(".link").data(json.links).enter().append("line").attr("class", "link");var node = svg.selectAll(".node").data(json.nodes).enter().append("g").attr("class", "node").call(force.drag); Now, we define the node as circles of size 6 and include the labels of each node. node.append("circle").attr("r", 6);node.append("text").attr("dx", 12).attr("dy", ".35em").text(function(d) { return d.name }); Finally, with the function, tick, run step-by-step the force layout simulation. force.on("tick", function(){link.attr("x1", function(d) { return d.source.x; }).attr("y1", function(d) { return d.source.y; }).attr("x2", function(d) { return d.target.x; }).attr("y2", function(d) { return d.target.y; });node.attr("transform", function(d){return "translate(" + d.x + "," + d.y + ")";})});});</script> In the image below we can see the result of the visualization. In order to run the visualization we just need to open a Command Terminal and run the following Python command or any other web server. >>python –m http.server 8000 Then you just need to open a web browser and type the direction http://localhost:8000/ForceGraph.html. In the HTML page we can see our Facebook graph with a gravity effect and we can interactively drag-and-drop the nodes. All the codes and datasets of this article may be found in the author github repository in the link below.https://github.com/hmcuesta/PDA_Book/tree/master/Chapter10 Summary In this article we developed our own social graph visualization tool with D3js, transforming the data obtained from Netvizz with GDF format into JSON. Resources for Article: Further resources on this subject: GNU Octave: Data Analysis Examples [Article] Securing data at the cell level (Intermediate) [Article] Analyzing network forensic data (Become an expert) [Article]
Read more
  • 0
  • 0
  • 8406

article-image-python-data-persistence-using-mysql
Packt
23 Oct 2009
8 min read
Save for later

Python Data Persistence using MySQL

Packt
23 Oct 2009
8 min read
To keep things simple though, the article doesn’t discuss how to implement database-backed web pages with Python, concentrating only on how to connect Python with MySQL. Sample Application The best way to learn new programming techniques is to write an application that exercises them. This article will walk you through the process of building a simple Python application that interacts with a MySQL database. In a nutshell, the application picks up some live data from a web site and then persists it to an underlying MySQL database. For the sake of simplicity, it doesn’t deal with a large dataset. Rather, it picks up a small subset of data, storing it as a few rows in the underlying database. In particular, the application gets the latest post from the Packt Book Feed page available at http://feeds.feedburner.com/packtpub/sDsa?format=xml. Then, it analyzes the post’s title, finding appropriate tags for the article associated with the post, and finally inserts information about the post into the posts and posttags underlying database tables. As you might guess, a single post may be associated with more than one tag, meaning a record in the posts table may be related to several records in the posttags table. Diagrammatically, the sample application components and their interactions might look like this: Note the use of appsample.py. This script file will contain all the application code written in Python. In particular, it will contain the list of tags, as well as several Python functions packaging application logic. Software Components To build the sample discussed in the article you’re going to need the following software components installed on your computer: Python 2.5.x MySQLdb 1.2.x MySQL 5.1 All these software components can be downloaded and used for free. Although you may already have these pieces of software installed on your computer, here’s a brief overview of where you can obtain them. You can download an appropriate Python release from the Downloads page at Python’s web site at http://python.org/download/. You may be tempted to download the most recent release. Before you choose the release, however, it is recommended that you visit the Python for MySQL page at http://sourceforge.net/projects/mysql-python/ to check what Python releases are supported by the current MySQLdb module that will be used to connect your Python installation with MySQL. MySQLdb is the Python DB API-2.0 interface for MySQL. You can pick up the latest MySQLdb package (version 1.2.2 at the time of writing) from the sourceforge.net’s Python for MySQL page at http://sourceforge.net/projects/mysql-python/. Before you can install it, though, make sure you have Python installed in your system. You can obtain the MySQL 5.1 distribution from the mysql.com web site at http://dev.mysql.com/downloads/mysql/5.1.html, picking up the package designed for your operating system. Setting up the Database Assuming you have all the software components that were outlined in the preceding section installed in your system, you can now start building the sample application. The first step is to create the posts and posttags tables in your underlying MySQL database. As mentioned earlier, a single post may be associated with more than one tag. What this means in practice is that the posts and posttags tables should have a foreign key relationship. In particular, you might create these tables as follows: CREATE TABLE posts ( title VARCHAR(256) PRIMARY KEY, guid VARCHAR(1000), pubDate VARCHAR(50) ) ENGINE = InnoDB; CREATE TABLE posttags ( title VARCHAR(256), tag VARCHAR(20), PRIMARY KEY(title,tag), FOREIGN KEY(title) REFERENCES posts(title) ) ENGINE = InnoDB; As you might guess, you don’t need to populate above tables with data now. This will be automatically done later when you launch the sample. Developing the Script Now that you have the underlying database ready, you can move on and develop the Python code to complete the sample. In particular, you’re going to need to write the following components in Python: tags nested list of tags that will be used to describe the posts obtained from the Packt Book Feed page. obtainPost function that will be used to obtain the information about the latest post from the Packt Book Feed page. determineTags function that will determine appropriate tags to be applied to the latest post obtained from the Packt Book Feed page. insertPost function that will insert the information about the post obtained into the underlying database tables: posts and posttags. execPr function that will make calls to the other, described above functions. You will call this function to launch the application. All the above components will reside in a single file, say, appsample.py that you can create in your favorite text editor, such as vi or Notepad. First, add the following import declarations to appsample.py: import MySQLdb import urllib2 import xml.dom.minidom As you might guess, the first module is required to connect Python with MySQL, providing the Python DB API-2.0 interface for MySQL. The other two are needed to obtain and then parse the Packt Book Feed page’s data. You will see them in action in the obtainPost function in a moment. But first let’s create a nested list of tags that will be used by the determineTags function that determines the tags appropriate for the post being analyzed. To save space here, the following list contains just a few tags. You may and should include more tags to this list, of course. tags=["Python","Java","Drupal","MySQL","Oracle","Open Source"] The next step is to add the obtainPost function responsible for getting the data from the Packt Book Feed page and generating the post dictionary that will be utilized in further processing: def obtainPost(): addr = "http://feeds.feedburner.com/packtpub/sDsa?format=xml" xmldoc = xml.dom.minidom.parseString(urllib2.urlopen(addr).read()) item = xmldoc.getElementsByTagName("item")[0] title = item.getElementsByTagName("title")[0].firstChild.data guid = item.getElementsByTagName("guid")[0].firstChild.data pubDate = item.getElementsByTagName("pubDate")[0].firstChild.data post ={"title": title, "guid": guid, "pubDate": pubDate} return post Now that you have obtained all the required information about the latest post on the Packt Book Feed page, you can analyze the post’s title to determine appropriate tags. For that, add the determineTags function to appsample.py: def determineTags(title, tagslist): curtags=[] for curtag in tagslist: if title.find(curtag)>-1:curtags.append(curtag) return curtags By now, you have both the post and tags to be persisted to the database. So, add the insertPost function that will handle this task (don’t forget to change the parameters specified to the MySQLdb.connect function for the actual ones): def insertPost(title, guid, pubDate, curtags): db=MySQLdb.connect(host="localhost",user="usrsample",passwd="pswd",db="dbsample") c=db.cursor() c.execute("""INSERT INTO posts (title, guid, pubDate) VALUES(%s, %s,%s)""", (title, guid, pubDate)) db.commit() for tag in curtags: c.execute("""INSERT INTO posttags (title, tag) VALUES(%s,%s)""", (title, tag)) db.commit() db.close() All that is left to do is add the execPr function that brings all the pieces together, calling the above functions in the proper order: def execPr(): p = obtainPost() t = determineTags(p["title"],tags) insertPost(p["title"], p["guid"], p["pubDate"], t) Now let’s test the code we just wrote. The simplest way to do this is through Python’s interactive command line. To start an interactive Python session, you can type python at your system shell prompt. It’s important to realize that since the sample discussed here is going to obtain some data from the web, you must connect to the Internet before you launch the application. Once you’re connected, you can launch the execPr function in your Python session, as follows: >>>import appsample >>>appsample.execPr() If everything is okay, you should see no messages. To make sure that everything really went as planned, you can check the posts and posttags tables. To do this, you might connect to the database with the MySQL command-line tool and then issue the following SQL commands: SELECT * FROM posts; The above should generate the output that might look like this: |title |guid |pubDate ------------------------------------------------------------------ Open Source CMS Award Voting Now Closed | http://www.packtpub.com/ article/2008-award-voting-closed | Tue, 21 Oct 2008 09:29:54 +0100 Then, you might want to check out the posttags table: SELECT * FROM posttags; This might generate the following output: |title |tag Open Source CMS Award Voting Now Closed | Open Source Please note that you may see different results since you are working with live data. Another thing to note here is that if you want to re-run the sample, you first need to empty the posts and posttags tables. Otherwise, you will encounter the problem related to the primary key constraints. However, that won’t be a problem at all if you re-run the sample in a few days, when a new post or posts appear on the Packt Book Feed page. Conclusion In this article you looked at a simple Python application persisting data to an underlying MySQL database. Although, for the sake of simplicity, the sample discussed here doesn’t offer a web interface, it illustrates how you can obtain data from the Internet, and then utilize it within your application, and finally store that data in the database.
Read more
  • 0
  • 0
  • 8403

article-image-introduction-to-llama
Dario Radečić
02 Jun 2023
7 min read
Save for later

Introduction to LLaMA

Dario Radečić
02 Jun 2023
7 min read
It seems like everyone, and their grandmothers, are discussing Large Language Models (LLMs) these days. These models got all the hype since ChatGPT's release in late 2022. The average user might get lost in acronyms such as GPT, PaLM, or LLaMA, and that’s understandable. This article will shed some light on why you should generally care about LLMs and exactly what they bring to the table. By the end of this article, you’ll have a fundamental understanding of the LLaMA model, how it compares to other large language models, and will have the 7B flavor of LLaMA running locally on your machine. There’s no time to waste, so let’s dive straight in! The Purpose of LLaMA and Other Large Language Models The main idea behind LLMs is to understand and generate human-like text based on the input you feed into them. Ask a human-like question and you’ll get a human-like response back. You know what we’re talking about if you’ve ever tried ChatGPT. These models are typically trained on huge volumes of data, sometimes even as large as everything that has been written on the Internet over some time span. This data is then fed into the algorithms using unsupervised learning which has the task of learning words and relationships between them. Large Language Models can be generic or domain-specific. You can use a generic LLM and fine-tune it for a certain task, similar to what OpenAI did with Codex (LLM for programming).As the end-user, you can benefit from LLMs in several ways:Content generation – You can use LLMs to generate content for personal or professional purposes, such as articles, emails, social media posts, and so on.Information retrieval – LLMs help you find relevant information quickly and often do a better job when compared to a traditional web search. Just be aware of the training date cap the model has – it might not do as well on the recent events.Language assistance and translation – These models can detect spelling errors and grammar mistakes, suggest writing improvements, provide synonyms, idioms, and even provide a meaningful translation from one language to another.At the end of the day, probably everyone can find a helpful use case in a large language model.But which one should you choose? There are many publicly available models, but the one that stands out recently is LLaMA. Let’s see why and how it works next. What is LLaMA and How it Works? LLaMA stands for “Large Language Model Meta AI” and is a large language model published by – you’ve guessed it – Meta AI. It was released in February 2023 in a variety of flavors – from 7 billion to 65 billion parameters.A LLaMA model uses the Transformer architecture and works by generating probability distributions over sequences of words (or tokens). In plain English, this means the LLaMA model predicts the next most reasonable word given the sequence of input words.It’s interesting to point out that LLaMA-13B (13 billion parameters) outperforms GPT-3 on most benchmarks, even though GPT-3 has 13 times more parameters (175 billion). The more parameter-rich LLaMA (65B parameters) is on par with the best large language models we have available today, according to the official paper by Meta AI.In fact, let’s take a look at these performance differences by ourselves. The following table from the official paper summarizes it well: Figure 1 - LLaMA performance comparison with other LLMs Generally speaking, the more parameters the LLaMA model contains, the better it performs. The interesting fact is that even the 7B version is comparable in performance – or even outperforms – the models with significantly more parameters. The 7B model performs reasonably well, so how can you try it out? In the next section, you’ll have LLaMA running locally with only two shell commands. How to Run LLaMA Locally? You’ll need a couple of things to run LLaMA locally – decent hardware (doesn’t have to be the newest), a lot of hard drive space, and a couple of software dependencies installed. It doesn’t matter which operating system you’re using, as the implementation we’re about to show you is cross-platform.For reference, we ran the 7B parameter model on an M1 Pro MacBook with 16 GB of RAM. The model occupied 31 GB of storage, and you can expect this amount to grow if you choose a LLaMA flavor with more parameters.Regarding software dependencies, you’ll need a recent version of Node. We used version 18.16.0 with  npm version 9.5.1.Once you have Node installed, open up a new Terminal/CMD window and run the following command. It will install the 7B LLaMA model: npx dalai llama install 7B You might get a prompt to install  dalai first, so just type  y into the console. Once Dalai is installed, it will proceed to download the model weights. You should see something similar during this process: Figure 2 - Downloading LLaMA 7B model weights It will take some time, depending on your Internet speed. Once done, you’ll have the 7B model available in the Dalie web UI. Launch it with the following shell command:npx dalai serve This is the output you should see: Figure 3 - Running dalai web UI locally The web UI is now running locally on port 3000. As soon as you open http://localhost:3000, you’ll be presented with the interface that allows you to choose the model, tweak the parameters, and select a prompting template.For reference, we’ve selected the chatbot template and left every setting as default. The prompt we’ve entered is “What is machine learning?” Here’s what the LLaMA model with 7B parameters outputted:  Figure 4 - Dalai user interface The answer is mostly correct, but the LLaMA response started looking like a blog post toward the end (“In this article…”). As with all large language models, you can use it to draw insights, but only after some human intervention.And that’s how you can run a large language model locally! Let’s make a brief recap next. ConclusionIt’s getting easier and cheaper to train large language models, which means the number of options you’ll have is only going to grow over time.LLaMA was only recently released to the public, and today you’ve learned what it is, got a high-level overview of how it works, and how to get it running locally. You might want to tweak the 7B version if you’re not getting the desired response or opt for a version with more parameters (if your hardware allows it). Either way, have fun!Author Bio:Dario Radečić is a Senior Data Scientist at Neos, Croatia. Book author: "Machine Learning Automation with TPOT".  Owner of betterdatascience.com. You can follow him on Medium: https://medium.com/@radecicdario
Read more
  • 0
  • 0
  • 8401

article-image-downloading-evolved-metalink
Packt
19 Mar 2010
7 min read
Save for later

Downloading evolved with Metalink

Packt
19 Mar 2010
7 min read
Downloading evolved with Metalink Anyone who has tried downloading a popular software or Linux distribution on or just after release day, knows of the pain of the phrase "connection timed out". Getting the software can be quite a struggle, despite all the mirrors and BitTorrent Samaritans. Anthony Bryan's Metalink is an open standard that makes downloading easier, faster, and more reliable by helping users extract the last drop of juice out of their connection. I emailed a set of questions to Bryan to understand what separates Metalink from other run-of-the-mill download accelerators and listen in awe as he explains how Metalink combines traditional HTTP and FTP methods of downloading files along with BitTorrent. Mayank Sharma:  Let's begin with the traditional what is question -- what is Metalink? Is it another way to download "stuff"? Anthony Bryan: Yep, Metalink is a way to download "stuff." But it's not a new transfer protocol like FTP or P2P method. It attempts to make the traditional download process simpler by automating advanced features and hiding complexity where possible. It's something that download programs like Web browsers, FTP programs, download managers, and P2P clients support. Information about the files to be downloaded is stored in a simple XML file with a .metalink extension. MS: So how exactly is using a metalink beneficial over other traditional methods of downloading? AB: In general, it should be easier and be error free. It can also be much faster. The information a download program would need is in a machine-processable format in the .metalink file, stuff like all the different ways to download a file (from multiple mirrors to P2P) and other things like the priority and geographical location of the mirrors, checksums, publisher, license, OS, language, etc. More the information, more the things that can be done! So, instead of a single link to a file, you have many links. This means higher availability. If many servers are down or very busy (like on a release day), your download program can automatically check links and see which ones are good. You don't have to manually gather and check each link. This improves usability. Some programs can use the multiple links to download different parts or segments of a file from many places, which is called multi-source downloading. Usually, your download will be much faster, but that all depends on your connection. A single source download usually can't offer as fast download speeds. Metalinks can also contain checksums or repair information to fix corrupted downloads, either files downloaded with other programs or repairing the download in progress. You can repair downloads with rsync and some P2P, but with Metalink you can do it over regular FTP/HTTP. These benefits are mainly seen in large downloads like ISO files (CD/DVD images), which can range from 500 MB to 4 GB or larger. Time is money and sometimes you need things downloaded before you can get work done. If there was an error in one of these files, it could be painful to need to re-download it. This could be very helpful to movie and music downloads, where (once this technology is included in browsers) you'll be able to download huge files and recover from errors, and download a whole album or multiple albums in one click, within a browser and not some special application. For instance, when you buy a bunch of albums on Bleep.com, you can download the files individually or in an archive. The archive takes up resources on the server and on your computer. With Metalink, you can list all the files and have them added to a download queue in one click. This is also useful for software like KDE that is available in multiple files. MS: Whoa, those are some advantages. While it's good for the person downloading, what about the person offering the download? Any reasons why he should make a .metalink for his downloads? AB: As we've already discussed, Metalink should offer your downloaders an improved experience - more reliable and without errors. This translates to cheaper bandwidth bills and support costs. Not very exciting unfortunately, but something that needs to be done since it's simple and can be automated. You can give different mirrors or P2P sources a priority. If you list FTP/HTTP mirrors and P2P, the mirrors can be used in case the person can't access P2P or if the file is no longer on the P2P network. This can be handy in situations where a business or university may block P2P because of copyright concerns or that they don't want someone continuously uploading and using bandwidth, or because of configuration issues with routers, etc. In the end, you want people to be able to download whatever you are distributing. MS: Great! So how does one create a .metalink for their downloads? AB: The easiest way to create a .metalink is with the Metalink Editor which has a graphical interface. If you're going to be making a bunch, you may want to use the automated tools which have a command line interface or server tools which are meant to be run on mirrors. This article on osresources details the non-graphical methods for .metalink creation. MS: Right. So do I need a special application to use a metalink download? Won't it work with wget? AB: Right now you do need a special application that supports reading the .metalink file and using that info. Luckily, most download managers support it right now, but no Web browsers natively. Support in Opera and Firefox would make the benefits way more accessible to regular people. wget doesn't support it yet, but aria2 is a similar command-line program that does. The beauty of free and open source software is that if people are interested enough, then support can be added to a bunch of download programs quickly and easily. MS: Do you track the number of projects using Metalink? AB: Yes, mostly free and open source projects use it right now, but some proprietary companies and game makes use it. OpenOffice.org, openSUSE, cURL, Arch Linux, DesktopBSD, blag linux, StartCom Linux, Berry Linux, PC-BSD, Linux Mint, Ubuntu Christian Edition, redWall Firewall, GoboLinux, TrueBSD, PuppyLinux, UniProt Consortium, Eiffel Software, and Ankama games have used Metalink. MS:  Any new feature Metalink users, both offering and downloading stuff, should keep an eye out for? AB: I view Metalink as basic infrastructure and hopefully it will get to the point where people don't even know they are using it. Metalink as it is now is Wave One (or maybe .5) of coming improvements. I think we can build on it and offer some really cool things, which will again work transparently for people. Not all clients support all the features of Metalink, like using repair information - that's a really important feature. Phex is the first program with Metalink support that is primarily P2P (other BitTorrent clients like aria2 and GetRight support it). You'll be able to export your library, or parts of it. If you want to share something with a friend, you can just email a .metalink or put it on a server for download, and they'll get the same exact files since they're identified with checksums. No weeding through search results searching for similar files will be needed. MS: Thanks for taking time out for the interview. Please share a little information about yourself and your life outside of computers. AB: I'm 29, just finished my Master's degree, and looking for work related to Metalink with open source projects.  :)  I live in southeastern Florida, in Pompano Beach. I'm pretty laid back and I love the ocean. I'm really into music, mostly instrumental, from electronic to Indian, dub, jazz, and even normal stuff. I like old books and movies and I'm interested in technology and how it affects our lives.
Read more
  • 0
  • 0
  • 8397

article-image-setting-citrix-components
Packt
03 Nov 2015
4 min read
Save for later

Setting Up the Citrix Components

Packt
03 Nov 2015
4 min read
In this article by Sunny Jha, the author of the book Mastering XenApp, we are going to implement the Citrix XenApp infrastructure components, which are going to work together to deliver the applications. The components we will be implementing are as follows: Setting up Citrix License Server Setting up Delivery Controller Setting up Director Setting up StoreFront Setting up Studio Once you will complete this article, you will be able to understand how to install the Citrix XenApp infrastructure components for the effective delivery of applications. (For more resources related to this topic, see here.) Setting up the Citrix infrastructure components You must be aware of the fact that Citrix reintroduced Citrix XenApp in the version of Citrix XenApp 7.5 with the new FMA-based architecture, replacing IMA. In this article, we will be setting up different Citrix components so that they can deliver the applications. As this is the proof of concept, I will be setting up almost all the Citrix components on the single Microsoft Windows 2012 R2 machine, where it is recommended that in the production environment, you should keep the Citrix components such as License Server, Delivery Controller, and StoreFront. These need to be installed on the separate servers to avoid the single point of failure and better performance. The components that we will be setting up in this article are: Delivery Controller: This Citrix component will act as broker, and the main function is to assign users to a server, based on their selection of application published. License Server: This will assign the license to the Citrix components as every Citrix product requires license in order to work. Studio: This will act as control panel for Citrix XenApp 7.6 delivery. Inside Citrix, studio administrator makes all the configuration and changes. Director: This component is basically for monitoring and troubleshooting, which is web-based application. StoreFront: This is the frontend of the Citrix infrastructure by which users connect to their application, either via receiver or web based. Installing of Citrix components In order to start the installation, we need the Citrix XenApp 7.6 DVD or ISO image. You can always download, from the Citrix website, all you need to have in the MyCitrix account. Follow these steps: Mount the disc/ISO you have downloaded. When you will double-click on the mounted disc, it will bring up a nice screen where you have to make the selection between XenApp Deliver applications or XenDesktop Deliver application and desktops: Once you have made the selection, it will show you the next option related to the product. Here, we need to select XenApp. Choose Delivery Controller from the options: The next screen will show you the License Agreement. You can go through it and accept the terms and click on Next: As described earlier, this is the proof of concept. We will install all the components on single server, but it is recommended to put each component on different server for better performance. Select all the components and click on Next: The next screen will show you the features that can be installed. As we have already installed the SQL server, we don't have to select the SQL Express, but we will choose Install Windows Remote Assistance. Click on Next: The next screen will show you the firewall ports that needs to be allowed to communicate, and it can be adjusted by Citrix as well. Click on Next: The next screen will show you the summary of your selection. Here, you can review your selection and click on Install to install the components: After you click on Install, it will go through the installation procedure, and once the installation is complete, click on Next. By following these steps, we completed the installation of the Citrix components such as Delivery Controller, Studio, Director, and StoreFront. We also adjusted the firewall ports as per the Citrix XenApp requirement. Summary In this article, you learned about setting up the Citrix infrastructure components and also how to install Citrix Director, License Server, Citrix Studio, and Citrix Director, and Citrix StoreFront. Resources for Article: Further resources on this subject: Getting Started – Understanding Citrix XenDesktop and its Architecture [article] High Availability, Protection, and Recovery using Microsoft Azure [article] A Virtual Machine for a Virtual World [article]
Read more
  • 0
  • 0
  • 8390
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-configuring-placeholder-datastores
Packt
20 May 2014
1 min read
Save for later

Configuring placeholder datastores

Packt
20 May 2014
1 min read
(For more resources related to this topic, see here.) Assuming that each of these paired sites is geographically separated, each site will have its own placeholder datastore. The following figure shows the site and placeholder datastore relationship: This is how you configure placeholder datastores: Navigate to vCenter Server's inventory home page and click on Site Recovery. Click on Sites in the left pane and select a site. Navigate to the Placeholder Datastores tab and click on Configure Placeholder Datastore, as shown in the following screenshot: In the Configure Placeholder Datastore window, select an appropriate datastore and click on OK. To confirm the selection, exit the window. Now, the Placeholder Datastores tab should show the configured placeholder. Refer to the following screenshot: If you plan to configure a Failback, repeat the procedure in the recovery site. Summary In this article, we covered the steps to be followed in order to configure placeholder datastores. Resources for Article: Further resources on this subject: Disaster Recovery for Hyper-V [Article] VMware vCenter Operations Manager Essentials - Introduction to vCenter Operations Manager [Article] Disaster Recovery Techniques for End Users [Article]
Read more
  • 0
  • 0
  • 8389

article-image-managing-pools-desktops
Packt
07 Oct 2015
14 min read
Save for later

Managing Pools for Desktops

Packt
07 Oct 2015
14 min read
In this article by Andrew Alloway, the author of VMware Horizon View High Availability, we will review strategies for providing High Availability for various types of VMware Horizon View desktop pools. (For more resources related to this topic, see here.) Overview of pools VMware Horizon View provides administrators with the ability to automatically provision and manage pools of desktops. As part of our provisioning of desktops, we must also consider how we will continue service for the individual users in the event of a host or storage failure. Generally High Availability requirements fall into two categories for each pool. We can have stateless desktops where the user information is not stored on the VM between sessions and Stateful desktops where the user information is stored on the desktop between sessions. Stateless desktops In a stateless configuration, we are not required to store data on the Virtual Desktops between user sessions. This allows us to use Local Storage instead of shared storage for our HA strategies as we can tolerate host failures without the use of shared disk. We can achieve a stateless desktop configuration using roaming profiles and/or View Persona profiles. This can greatly reduce cost and maintenance requirements for View Deployments. Stateless desktops are typical in the following environments: Task Workers: A group of workers where the tasks are well known and they all share a common set of core applications. Task workers can use roaming profiles to maintain data between user sessions. In a multi shift environment, having stateless desktops means we only need to provision as many desktops that will be used consecutively. Task Worker setups are typically found in the following scenarios: Data entry Call centers Finance, Accounts Payables, Accounts Receivables Classrooms (in some situations) Laboratories Healthcare terminals Kiosk Users: A group of users that do not login. Logins are typically automatic or without credentials. Kiosk users are typically untrusted users. Kiosk VMs should be locked down and restricted to only the core applications that need to be run. Kiosks are typically refreshed after logoff or at scheduled times after hours. Kiosks can be found in situations such as the following: Airline Check-In stations Library Terminals Classrooms (in some situations) Customer service terminals Customer Self-Serve Digital Signage Stateful desktops Statefull desktops have some advantages from reduced iops and higher disk performance due to the ability to choose thick provisioning. Stateful desktops are desktops that require user data to be stored on the VM or Desktop Host between user sessions. These machines typically are required by users who will extensively customize their desktop in non-trivial ways, require complex or unique applications that are not shared by a large group or require the ability to modify their VM Stateful Desktops are typically used for the following situations: Users who require the ability to modify the installed applications Developers IT Administrators Unique or specialized users Department Managers VIP staff/managers Dedicated pools Dedicated pools are View Desktops provisioned using thin or thick provisioning. Dedicated pools are typically used for Stateful Desktop deployments. Each desktop can be provisioned with a dedicated persistent disk used for storing the User Profile and data. Once assigned a desktop that user will always log into the same desktop ensuring that their profile is kept constant. During OS refresh, balances and recomposes the OS disk is reverted back to the base image. Dedicated Pools with persistent disks offer simplicity for managing desktops as minimal profile management takes place. It is all managed by the View Composer/View Connection Server. It also ensures that applications that store profile data will almost always be able to retrieve the profile data on the next login. Meaning that the administrator doesn't have to track down applications that incorrectly store data outside the roaming profile folder. HA considerations for dedicated pools Dedicated pools unfortunately have very difficult HA requirements. Storing the user profile with the VM means that the VM has to be stored and maintained in an HA aware fashion. This almost always results in a shared disk solution being required for Dedicated Pools. In the event of a host outage other hosts connected to the same storage can start up the VM. For shared storage, we can use NFS, iSCSI, Fibre Channel, or VMware Virtual SAN storage. Consider investing in storage systems with primary and backup controllers as we will be dependent on the disk controllers being always available. Backups are also a must with this system as there is very little recovery options in the event of a storage array failure. Floating Pools Floating pools are a pool of desktops where any user can be assigned to any desktop in the pool upon login. Floating pools are generally used for stateless desktop deployments. Floating pools can be used with roaming profiles or View Persona to provide a consistent user experience on login. Since floating pools are treated as disposable VMs, we open up additional options for HA. Floating pools are given 2 local disks, the OS disk which is a replica from the assigned base VM, and the Disposable Disk where the page file, hibernation file, and temp drive are located. When Floating pools are refreshed, recomposed or rebalanced, all changes made to the desktop by the users are lost. This is due to the Disposable Disk being discarded between refreshes and the OS disk being reverted back to the Base Image. As such any session information such as Profile, Temp directory, and software changes are lost between refreshes. Refreshes can be scheduled to occure after logoff, after every X days or can be manually refreshed. HA considerations for floating pools Floating pools can be protected in several ways depending on the environment. Since floating pools can be deployed on local storage we can protect against a host failure by provisioning the Floating Pool VMs on multiple separate hosts. In the event of a host failure the remaining Virtual Desktops will be used to log users in. If there is free capacity in the cluster more Virtual Desktops will be provisioned on other hosts. For environments with shared storage Floating Pools can still be deployed on the shared storage but it is a good idea to have a secondary shared storage device or a highly available storage device. In the event of a storage failure the VMs can be started on the secondary storage device. VMware Virtual SAN is inherently HA safe and there is no need for a secondary datastore when using Virtual SAN. Many floating pool environments will utilize a profile management solution such as Roaming Profiles or View Persona Management. In these situations it is essential to setup a redundant storage location for View Profiles and or Roaming Profiles. In practice a Windows DFS share is a convenient and easy way to guard profiles against loss in the event of an outage. DFS can be configured to replicate changes made to the profile in real time between hosts. If the Windows DFS server is provisioned as VMs on shared storage make sure to create a DRS rule to separate the VMs onto different hosts. Where possible DFS servers should be stored on separate disk arrays to ensure they data is preserved in the event of the Disk Array, or Storage Processor failure. For more information regarding Windows DFS you can visit the link below https://technet.microsoft.com/en-us/library/jj127250.aspx Manual pools Manual pools are custom dedicated desktops for each user. A VM is manually built for each user who is using the manual pool. Manual Pools are Stateful pools that generally do not utilize profile management technologies such as View Persona or Roaming Profiles. Like Dedicated pools once a user is assigned to a VM they will always log into the same VM. As such HA requirements for manual pools are very similar to dedicated pools. Manual desktops can be configured in almost any maner desired by the administrator. There are no requirements for more than one disk to be attached to the Manual Pool desktop. Manual pools can also be configured to utilize physical hardware as the Desktop such as Blade Servers, Desktop Computers or even Laptops. In this situation there are limited high availability options without investing in exotic and expensive hardware. As best practice the physical hosts should be built with redundant power supplies, ECC RAM, mirrored hard disks pending budget and HA requirements. There should be a good backup strategy for managing physical hosts connected to the Manual Pools. HA considerations for manual pools Manual pools like dedicated pools have a difficult HA requirement. Storing the user profile with the VM means that the VM has to be stored and maintained in an HA aware fashion. This almost always results in a shared disk solution being required for Manual Pools. In the event of a host outage other hosts connected to the same storage can start up the VM. For shared storage, we can use NFS, iSCSI, Fibre Channel, or VMware VSAN storage. Consider investing in storage systems with primary and backup controllers as we will be dependent on the disk controllers being always available. Backups are also a must with this system as there is very little recovery options in the event of a storage array failure. VSAN deployments are inherently HA safe and are excellent candidates for Manual Pool storage. Manual pools given their static nature also have the option of using replication technology to backup the VMs onto another disk. You can use VMware vSphere Replication to do automatic replication or use a variety of storage replication solutions offered by storage and backup vendors. In some cases it may be possible to use fault tolerance on the Virtual Desktops for truly high availability. Note that this would limit the individual VMs to a single vCPU which may be undesirable. Remote Desktop services pools Remote Desktop Services Pools (RDS Pools) are pools where the remote session or application is hosted on a Windows Remote Desktop Server. The application or remote session is run under the users' credentials. Usually all the user data is stored locally on the Remote Desktop Server but can also be stored remotely using Roaming Profiles or View Persona Profiles. Folder Redirection to a central network location is also used with RDS Pools. Typical uses for Remote Desktop Services is for migrating users off legacy RDS environments, hosting applications, and providing access to troublesome applications or applications with large memory foot prints. The Windows Remote Desktop Server can be either a VM or a standalone physical host. It can be combined with Windows Clustering technology to provide scalability and high availability. You can also deploy a load balancer solution to manage connections between multiple Windows Remote Desktop Servers. Remote Desktop services pool HA considerations Remote Desktop services HA revolves around protecting individual RDS VMs or provisioning a cluster of RDS servers. When a single VM is deployed wilth RDS generally it is best to use vSphere HA and clustering featurs to protect the VM. If the RDS resources are larger than practical for a VM then we must focus on protecting individual host or clustering multiple hosts. When the Windows Remote Desktop Server is deployed as a VM the following options are available: Protect the VM with VMware HA, using shared storage This allows vCenter to fail over the VM to another host in the event of a host failure. vSphere will be responcible for starting the VM on another host. The VM will resume from a crashed state. Replicate the Virtual Machine to separate disks on separate hosts using VMware Virtual SAN: Same as above but in this case the VM has been replicated to another host using Virtual SAN technology. The remote VM will be started up from a crashed state, using the last consistent harddrive image that was replicated. Using replication technologies such as vSphere Replication: The VM will be periodically synchronized to a remote host. In the event of a host failure we can manually activate the remotely synchronized VM. Use a Vendors Storage Level replication: In this case we allow our storage vendor to provide replication technology to provide a redundant backup. This protects us in the event of a storage or host failure. These failures can be automated or manual. Consult with your Storage Vendor for more information. Protect the VM using backup technologies: This provides redundancy in the sense that we won't loose the VM if it fails. Unfortuantely you are at the mercy of your restore process to bring the VM back to life. The VM will resume from a crashed state. Always keep backups of production servers. For RDS servers running on a dedicated server we could utilize the following: Redundant power supplies: Redundant power supplies will keep the server going while a PSU is being replaced or becomes defective. It is also a good idea to have 2 separate power sources for each power supply. Simple things like a power bar going faulty or triping a breaker could bring down the server if there are not two independent power sources. Uninteruptable Power Supply: Battery backups are always a must for production level equipment. Make sure to scale the UPS to provide adequate power and duration for your environment. Redundant network interfaces: In rare sucumstances a NIC can go bad or a cable can be damaged. In this case redundant NICs will prevent a server outage. Remember that to protect against a switch outage we should plug the NICs into separate switches. Mirrored or redundant disks: Harddrives are one of the most common failures in computers. Mirrored harddrives or RAID configurations are a must for production level equipment. 2 or more hosts: Clustering physical servers will ensure that host failures won't cause downtime. Consider multi site configurations for even more redundancy. Shared Strategies for VMs and Hardware: Provide High Availability to the RDS using Microsoft Network Load Balancer (NLB): Microsoft Network Load Balancer can provide load balancing to the RDS servers directy. In this situation the clients would connect to a single IP managed by the NLB which would randomly be assigned to a server. Provide High Availability using a load balancer to manage sessions between RDS servers: Using a hardware or software load balancer is can be used instead of Microsoft Network Load Balancers. Load Balancer vendors provide a high variety of capabilities and features for their load balancers. Consult your load balancer vendor for best practices. Use DNS Round Robin to alternate between RDS hosts: On of the most cost effective load balancing methods. It has the drawback of not being able to balance the load or to direct clients away from failed hosts. Updating DNS may delay adding new capacity to the cluster or delay removing a failed host from the cluster. Remote Desktop Connection Broker with High Availability: We can provide RDS failover using the Connection Broker feature of our RDS server. For more details see the link below. For more information regarding Remote Desktop Connection Broker with High Availability see: https://technet.microsoft.com/en-us/library/ff686148%28WS.10%29.aspx Here is an example topology using physical or virtual Microsoft RDS Servers. We use a load balancing technology for the View Connection Servers as described in the previous chapter. We then will connect to the RDS via either a load balancer, DNS round robin, or Cluster IP. Summary In this article, we covered the concept of stateful and stateless desktops and the consequences and techniques for supporting each in a highly available environment. Resources for Article: Further resources on this subject: Working with Virtual Machines[article] Storage Scalability[article] Upgrading VMware Virtual Infrastructure Setups [article]
Read more
  • 0
  • 0
  • 8387

article-image-preprocessing-data
Packt
16 Aug 2016
5 min read
Save for later

Preprocessing the Data

Packt
16 Aug 2016
5 min read
In this article, by Sampath Kumar Kanthala, the author of the book Practical Data Analysis discusses how to obtain, clean, normalize, and transform raw data into a standard format like CVS or JSON using OpenRefine. In this article we will cover: Data Scrubbing Statistical methods Text Parsing Data Transformation (For more resources related to this topic, see here.) Data scrubbing Scrubbing data also called data cleansing, is the process of correcting or removing data in a dataset that is incorrect, inaccurate, incomplete, improperly formatted, or duplicated. The result of the data analysis process not only depends on the algorithms, it depends on the quality of the data. That's why the next step after obtaining the data, is the data scrubbing. In order to avoid dirty data our dataset should possess the following characteristics: Correct Completeness Accuracy Consistency Uniformity Dirty data can be detected by applying some simple statistical data validation also by parsing the texts or deleting duplicate values. Missing or sparse data can lead you to highly misleading results. Statistical methods In this method we need some context about the problem (knowledge domain) to find values that are unexpected and thus erroneous, even if the data type match but the values are out of the range, it can be resolved by setting the values to an average or mean value. Statistical validations can be used to handle missing values which can be replaced by one or more probable values using Interpolation or by reducing the data set using decimation. Mean: Value calculated by summing up all values and then dividing by the number of values. Median: The median is defined as the value where 50% of values in a range will be below, 50% of values above the value. Range constraints: Numbers or dates should fall within a certain range. That is, they have minimum and/or maximum possible values. Clustering: Usually, when we obtain data directly from the user some values include ambiguity or refer to the same value with a typo. For example, "Buchanan Deluxe 750ml 12x01 "and "Buchanan Deluxe 750ml   12x01." which are different only by a "." or in the case of "Microsoft" or "MS" instead of "Microsoft Corporation" which refer to the same company and all values are valid. In those cases, grouping can help us to get accurate data and eliminate duplicated enabling a faster identification of unique values. Text parsing We perform parsing to help us to validate if a string of data is well formatted and avoid syntax errors. Regular expression patterns usually, text fields would have to be validated this way. For example, dates, e-mail, phone numbers, and IP address. Regex is a common abbreviation for "regular expression"): In Python we will use re module to implement regular expressions. We can perform text search and pattern validations. First, we need to import the re module. import re In the follow examples, we will implement three of the most common validations (e-mail, IP address, and date format). E-mail validation: myString = 'From: readers@packt.com (readers email)' result = re.search('([w.-]+)@([w.-]+)', myString) if result: print (result.group(0)) print (result.group(1)) print (result.group(2)) Output: >>> readers@packt.com >>> readers >>> packt.com The function search() scans through a string, searching for any location where the Regex matches. The function group() helps us to return the string matched by the Regex. The pattern w matches any alphanumeric character and is equivalent to the class [a-zA-Z0-9_]. IP address validation: isIP = re.compile('d{1,3}.d{1,3}.d{1,3}.d{1,3}') myString = " Your IP is: 192.168.1.254 " result = re.findall(isIP,myString) print(result) Output: >>> 192.168.1.254 The function findall() finds all the substrings where the Regex matches, and returns them as a list. The pattern d matches any decimal digit, is equivalent to the class [0-9]. Date format: myString = "01/04/2001" isDate = re.match('[0-1][0-9]/[0-3][0-9]/[1-2][0-9]{3}', myString) if isDate: print("valid") else: print("invalid") Output: >>> 'valid' The function match() finds if the Regex matches with the string. The pattern implements the class [0-9] in order to parse the date format. For more information about regular expressions: http://docs.python.org/3.4/howto/regex.html#regex-howto Data transformation Data transformation is usually related with databases and data warehouse where values from a source format are extract, transform, and load in a destination format. Extract, Transform, and Load (ETL) obtains data from data sources, performs some transformation function depending on our data model and loads the result data into destination. Data extraction allows us to obtain data from multiple data sources, such as relational databases, data streaming, text files (JSON, CSV, XML), and NoSQL databases. Data transformation allows us to cleanse, convert, aggregate, merge, replace, validate, format, and split data. Data loading allows us to load data into destination format, like relational databases, text files (JSON, CSV, XML), and NoSQL databases. In statistics data transformation refers to the application of a mathematical function to the dataset or time series points. Summary In this article, we explored the common data sources and implemented a web scraping example. Next, we introduced the basic concepts of data scrubbing like statistical methods and text parsing. Resources for Article:   Further resources on this subject: MicroStrategy 10 [article] Expanding Your Data Mining Toolbox [article] Machine Learning Using Spark MLlib [article]
Read more
  • 0
  • 0
  • 8387

article-image-creating-reports-using-sql-server-2016-reporting-services
Kunal Chaudhari
04 Jan 2018
6 min read
Save for later

Creating reports using SQL Server 2016 Reporting Services

Kunal Chaudhari
04 Jan 2018
6 min read
[box type="note" align="" class="" width=""]This article is an excerpt from a book authored by Dinesh Priyankara and Robert C. Cain, titled SQL Server 2016 Reporting Services Cookbook.This book will help you create cross-browser and cross-platform reports using SQL Server 2016 Reporting Services.[/box] In today’s tutorial, we explore steps to create reports on multiple axis charts with SQL Server 2016. Often you will want to have multiple items plotted on a chart. In this article, we will plot two values over time, in this case, the Total Sales Amount (Excluding Tax) and the Total Tax Amount. As you might expect though, the tax amounts are going to be a small percentage of the sales amounts. By default, this would create a chart with a huge gap in the middle and a Y Axis that is quite large and difficult to pinpoint values on. To prevent this, Reporting Services allows us to place a second Y Axis on our charts. With this article, we'll explore both adding a second line to our chart as well as having it plotted on a second Y-Axis. Getting ready First, we'll create a new Reporting Services project to contain it. Name this new project Chapter03. Within the new project, create a Shared Data Source that will connect to the WideWorldImportersDW database. Name the new data source after the database, WideWorldImportersDW. Next, we'll need data. Our data will come from the sales table, and we will want to sum our totals by year, so we can plot our years across the X-Axis. For the Y-Axis, we'll use the totals of two fields: TotalExcludingTax and TaxAmount. Here is the query by which we will accomplish this: SELECT YEAR([Invoice Date Key]) AS InvoiceYear ,SUM([Total Excluding Tax]) AS TotalExcludingTax ,SUM([Tax Amount]) AS TotalTaxAmount FROM [Fact].[Sale] GROUP BY YEAR([Invoice Date Key]) How to do it… Right-click on the Reports branch in the Solution Explorer. Go to Add | New Item… from the pop-up menu. On the Add New Item dialog, select Report from the choice of templates in the middle (do not select Report wizard). At the bottom, name the report Report 03-01 Multi Axis Charts.rdl and click on Add. Go to the Report Data tool pane. Right-click on Data Sources and then click Add Data Source… from the menu. In the Name: area, enter WideWorldImportersDW. Change the data source option to the Use shared dataset source reference. In the dropdown, select WideWorldImportersDW. Click on OK to close the DataSet Properties window. Right-click on the Datasets branch and select Add Dataset…. Name the dataset SalesTotalsOverTime. Select the Use a dataset embedded in my report option. Select WideWorldImportersDW in the Data source dropdown. Paste in the query from the Getting ready area of this article: When your window resembles that of the preceding figure, click on OK. Next, go to the Toolbox pane. Drag and drop a Chart tool onto the report. Select the leftmost Line chart from the Select Chart Type window, and click on OK. Resize the chart to a larger size. (For this demo, the exact size is not important. For your production reports, you can resize as needed using the Properties window, as seen previously.) Click inside the main chart area to make the Chart Data dialog appear to the right of the chart. Click on the + (plus button) to the right of the Values. Select TotalExcludingTax. Click on the plus button again, and now pick TotalTaxAmount. Click on the + (plus button) beside Category Groups, and pick InvoiceYear. Click on Preview. You will note the large gap between the two graphed lines. In addition, the values for the Total Tax Amount are almost impossible to guess, as shown in the following figure: Return to the designer, and again click in the chart area to make the Chart Data dialog appear. In the Chart Data dialog, click on the dropdown beside TotalTaxAmount: Select Series Properties…. Click on the Axes and Chart Area page, and for Vertical axis, select Secondary: Click on OK to close the Series Properties window. Right-click on the numbers now appearing on the right in the vertical axis area, and select Secondary Vertical Axis Properties in the menu. In the Axis Options, uncheck Always include zero. Click on the Number page. Under Category, select Currency. Change the Decimal places to 0, and place a check in Use 1000 separator. Click on OK to close this window. Now move to the vertical axis on the left-hand side of the chart, right-click, and pick Vertical Axis Properties. Uncheck Always include zero. On the Number page, pick Currency, set Decimal places to 0, and check Use 1000 separator. Click on OK to close. Click on the Preview tab to see the results: You can now see a chart with a second axis. The monetary amounts are much easier to read. Further, the plotted lines have a similar rise and fall, indicating the taxes collected matched the sales totals in terms of trending. SSRS is capable of plotting multiple lines on a chart. Here we've just placed two fields, but you can add as many as you need. But do realize that the more lines included, the harder the chart can become to read. All that is needed is to put the additional fields into the Values area of the Chart Data window. When these values are of similar scale, for example, sales broken up by state, this works fine. There are times though when the scale between plotted values is so great that it distorts the entire chart, leaving one value in a slender line at the top and another at the bottom, with a huge gap in the middle. To fix this, SSRS allows a second Y-Axis to be included. This will create a scale for the field (or fields) assigned to that axis in the Series Properties window. To summarize, we learned how creating reports with multiple axis is much more simpler with SQL Server 2016 Reporting Services. If you liked our post, check out the book SQL Server 2016 Reporting Services Cookbook to know more about different types of reportings and Power BI integrations.  
Read more
  • 0
  • 0
  • 8384
article-image-physics-uikit-dynamics
Packt
14 May 2014
8 min read
Save for later

Physics with UIKit Dynamics

Packt
14 May 2014
8 min read
(For more resources related to this topic, see here.) Motion and physics in UIKit With the introduction of iOS 7, Apple completely removed the skeuomorphic design that has been used since the introduction of the iPhone and iOS. In its place is a new and refreshing flat design that features muted gradients and minimal interface elements. Apple has strongly encouraged developers to move away from a skeuomorphic and real-world-based design in favor of these flat designs. Although we are guided away from a real-world look, Apple also strongly encourages that your user interface have a real-world feel. Some may think this is a contradiction; however, the goal is to give users a deeper connection to the user interface. UI elements that respond to touch, gestures, and changes in orientation are examples of how to apply this new design paradigm. In order to help assist in this new design approach, Apple has introduced two very nifty APIs, UIKit Dynamics and Motion Effects. UIKit Dynamics To put it simply, iOS 7 has a fully featured physics engine built into UIKit. You can manipulate specific properties to provide a more real-world feel to your interface. This includes gravity, springs, elasticity, bounce, and force to name a few. Each interface item will contain its own properties and the dynamic engine will abide by these properties. Motion effects One of the coolest features of iOS 7 on our devices is the parallax effect found on the home screen. Tilting the device in any direction will pan the background image to emphasize depth. Using motion effects, we can monitor the data supplied by the device's accelerometer to adjust our interface based on movement and orientation. By combining these two features, you can create great looking interfaces with a realistic feel that brings it to life. To demonstrate UIKit Dynamics, we will be adding some code to our FoodDetailViewController.m file to create some nice effects and animations. Adding gravity Open FoodDetailViewController.m and add the following instance variables to the view controller: UIDynamicAnimator* animator; UIGravityBehavior* gravity; Scroll to viewDidLoad and add the following code to the bottom of the method: animator = [[UIDynamicAnimator alloc] initWithReferenceView:self.view]; gravity = [[UIGravityBehavior alloc] initWithItems:@[self.foodImageView]]; [animator addBehavior:gravity]; Run the application, open the My Foods view, select a food item from the table view, and watch what happens. The food image should start to accelerate towards the bottom of the screen until it eventually falls off the screen, as shown in the following set of screenshots: Let's go over the code, specifically the two new classes that were just introduced, UIDynamicAnimator and UIGravityBehavior. UIDynamicAnimator This is the core component of UIKit Dynamics. It is safe to say that the dynamic animator is the physics engine itself wrapped in a convenient and easy-to-use class. The animator will do nothing on its own, but instead keep track of behaviors assigned to it. Each behavior will interact inside of this physics engine. UIGravityBehavior Behaviors are the core compositions of UIKit Dynamics animation. These behaviors all define individual responses to the physics environment. This particular behavior mimics the effects of gravity by applying force. Each behavior is associated with a view (or views) when created. Because you explicitly define this property, you can control which views will perform the behavior. Behavior properties Almost all behaviors have multiple properties that can be adjusted to the desired effect. A good example is the gravity behavior. We can adjust its angle and magnitude. Add the following code before adding the behavior to the animator: gravity.magnitude = 0.1f; Run the application and test it to see what happens. The picture view will start to fall; however, this time it will be at a much slower rate. Replace the preceding code line with the following line: gravity.magnitude = 10.0f; Run the application, and this time you will notice that the image falls much faster. Feel free to play with these properties and get a feel for each value. Creating boundaries When dealing with gravity, UIKit Dynamics does not conform to the boundaries of the screen. Although it is not visible, the food image continues to fall after it has passed the edge of the screen. It will continue to fall unless we set boundaries that will contain the image view. At the top of the file, create another instance variable: UICollisionBehavior *collision; Now in our viewDidLoad method, add the following code below our gravity code: collision = [[UICollisionBehavior alloc] initWithItems:@[self.foodImageView]]; collision.translatesReferenceBoundsIntoBoundary = YES; [animator addBehavior:collision]; Here we are creating an instance of a new class (which is a behavior), UICollisionBehavior. Just like our gravity behavior, we associate this behavior with our food image view. Rather than explicitly defining the coordinates for the boundary, we use the convenient translatesReferenceBoundsIntoBoundary property on our collision behavior. By setting this property to yes, the boundary will be defined by the bounds of the reference view that we set when allocating our UIDynamics animator. Because the reference view is self.view, the boundary is now the visible space of our view. Run the application and watch how the image will fall, but stop once it reaches the bottom of the screen, as shown in the following screenshot: Collisions With our image view responding to gravity and our screen bounds we can start detecting collisions. You may have noticed that when the image view is falling, it falls right through our two labels below it. This is because UIKit Dynamics will only respond to UIView elements that have been assigned behaviors. Each behavior can be assigned to multiple objects, and each object can have multiple behaviors. Because our labels have no behaviors associated with them, the UIKit Dynamics physics engine simply ignores it. Let's make the food image view collide with the date label. To do this, we simply need to add the label to the collision behavior allocation call. Here is what the new code looks like: collision = [[UICollisionBehavior alloc] initWithItems:@[self.foodImageView, self.foodDateLabel]]; As you can see, all we have done is add self.foodDateLabel to the initWithItems array property. As mentioned before, any single behavior can be associated with multiple items. Run your code and see what happens. When the image falls, it hits the date label but continues to fall, pushing the date label with it. Because we didn't associate the gravity behavior with the label, it does not fall immediately. Although it does not respond to gravity, the label will still be moved because it is a physics object after all. This approach is not ideal, so let's use another awesome feature of UIKit Dynamics, invisible boundaries. Creating invisible boundaries We are going to take a slightly different approach to this problem. Our label is only a point of reference for where we want to add a boundary that will stop our food image view. Because of this, the label does not need to be associated with any UIKit Dynamic behaviors. Remove self.foodDateLabel from the following code: collision = [[UICollisionBehavior alloc] initWithItems:@[self.foodImageView, self.foodDateLabel]]; Instead, add the following code to the bottom of viewDidLoad but before we add our collision behavior to the animator: // Add a boundary to the top edge CGPoint topEdge = CGPointMake(self.foodDateLabel.frame.origin.x + self.foodDateLabel.frame.size.width, self.foodDateLabel.frame.origin.y); [collision addBoundaryWithIdentifier:@"barrier" fromPoint:self.foodDateLabel.frame.origin toPoint:topEdge]; Here we add a boundary to the collision behavior and pass some parameters. First we define an identifier, which we will use later, and then we pass the food date label's origin as the fromPoint property. The toPoint property is set to the CGPoint we created using the food date label's frame. Go ahead and run the application, and you will see that the food image will now stop at the invisible boundary we defined. The label is still visible to the user, but the dynamic animator ignores it. Instead the animator sees the barrier we defined and responds accordingly, even though the barrier is invisible to the user. Here is a side-by-side comparison of the before and after: Dynamic items When using UIKit Dynamics, it is important to understand what UIKit Dynamics items are. Rather than referencing dynamics as views, they are referenced as items, which adhere to the UIDynamicItem protocol. This protocol defines the center, transform, and bounds of any object that adheres to this protocol. UIView is the most common class that adheres to the UIDynamicItem protocol. Another example of a class that conforms to this protocol is the UICollectionViewLayoutAttributes class. Summary In this article, we covered some basics of how UIKit Dynamics manages your application's behaviors, that enables us to create some really unique interface effects. Resources for Article: Further resources on this subject: Linking OpenCV to an iOS project [article] Unity iOS Essentials: Flyby Background [article] New iPad Features in iOS 6 [article]
Read more
  • 0
  • 0
  • 8384

article-image-building-app-using-backbonejs
Packt
29 Jul 2013
7 min read
Save for later

Building an app using Backbone.js

Packt
29 Jul 2013
7 min read
(For more resources related to this topic, see here.) Building a Hello World app For building the app you will need the necessary script files and a project directory laid out. Let's begin writing some code. This code will require all scripts to be accessible; otherwise we'll see some error messages. We'll also go over the Web Inspector and use it to interact with our applications. Learning this tool is essential for any web application developer to proficiently debug (and even write new code for) their application. Step 1 – adding code to the document Add the following code to the index.html file. I'm assuming the use of LoDash and Zepto, so you'll want to update the code accordingly: <!DOCTYPE HTML> <html> <head> <title>Backbone Application Development Starter</title> <!-- Your Utility Library --> <script src ="scripts/lodash-1.3.1.js"></script> <!-- Your Selector Engine --> <script src ="scripts/zepto-1.0.js"></script> <script src ="scripts/backbone-1.0.0.js"></script> </head> <body> <div id="display"> <div class ="listing">Houston, we have a problem.</div> </div> </body> <script src ="scripts/main.js"></script> </html> This file will load all of our scripts and has a simple <div>, which displays some content. I have placed the loading of our main.js file after the closing <body> tag. This is just a personal preference of mine; it will ensure that the script is executed after the elements of the DOM have been populated. If you were to place it adjacent to the other scripts, you would need to encapsulate the contents of the script with a function call so that it gets run after the DOM has loaded; otherwise, when the script runs and tries to find the div#display element, it will fail. Step 2 – adding code to the main script Now, add the following code to your scripts/main.js file: var object = {};_.extend(object, Backbone.Events);object.on("show-message", function(msg) {$('#display .listing').text(msg);});object.trigger("show-message", "Hello World"); Allow me to break down the contents of main.js so that you know what each line does. var object = {}; The preceding line should be pretty obvious. We're just creating a new object, aptly named object, which contains no attributes. _.extend(object, Backbone.Events); The preceding line is where the magic starts to happen. The utility library provides us with many functions, one of which is extend. This function takes two or more objects as its arguments, copies the attributes from them, and appends them to the first object. You can think of this as sort of extending a class in a classical language (such as PHP). Using the extend utility function is the preferred way of adding Backbone functionality to your classes. In this case, our object is now a Backbone Event object, which means it can be used for triggering events and running callback functions when an event is triggered. By extending your objects in this manner, you have a common method for performing event-based actions in your code, without having to write one-off implementations. object.on("show-message", function(msg) {$('#display .listing').text(msg);}); The preceding code adds an event listener to our object. The first argument to the on function is the name of the event, in this case show-message. This is a simple string that describes the event you are listening for and will be used later for triggering the events. There isn't a requirement as far as naming conventions go, but try to be descriptive and consistent. The second argument is a callback function, which takes an argument that can be set at the time the event is triggered. The callback function here is pretty simple; it just queries the DOM using our selector engine and updates the text of the element to be the text passed into the e vent. object.trigger("show-message", "Hello World"); Finally, we trigger our event. Simply use the trigger function on the object, tell it that we are triggering the show-message event, and pass in the argument for the trigger callback as the second argument. Step 3 – opening the project in your browser This will show us the text Hello World when we open our index.html file in our browser. Don't believe me? Double click on the file now, and you should see the following screen: If Chrome isn't your default browser, you should be able to right-click on the file from within your file browser and there should be a way to choose which application to open the file with. Chrome should be in that list if it was installed properly. Step 4 – encountering a problem Do you see something other than our happy Hello World message? The code is set up in a way that it should display the message Houston , we have a problem if something were to go wrong (this text is what is displayed in the DOM by default, before having JavaScript run and replace it). The missing script file The first place to look for the problem is the Network tab of the Web Inspector. This tab shows us each of the files that were downloaded by the browser to satisfy the rendering of the page. Sometimes, the Console tab will also tell us when a file wasn't properly downloaded, but not always. The following screenshot explains this: If you look at the Network tab, you can see an item clearly marked in red ( backbone-1.0.0.js ). In my project folder, I had forgotten to download the Backbone library file, thus the browser wasn't able to load the file. Note that we are loading the file from the local filesystem, so the status column says either Success or Failure . If these files were being loaded from a real web server, you would see the actual HTTP status codes, such as 200 OK for a successful file download or 404 Not Found for a missing file. The script typo Perhaps you have made a typo in the script while copying it from the book. If you have any sort of issues resulting from incorrect JavaScript, they will be visible in the Console tab, as shown in the following screenshot: In my example, I had forgotten the line about extending my object with the Backbone events class. Having left out the extend line of code caused the object to be missing some functionality, specifically the method on(). Notice how the console displays which filename and line number the error is on. Feel free to remove and add code to the files and refresh the page to get a feel for what the errors look like in the console. This is a great way to get a feel for debugging Backbone-based (and, really, any JavaScript) applications. Summary In this article we learned how to develop an app using Backbone.js. Resources for Article : Further resources on this subject: JBoss Portals and AJAX - Part 1 [Article] Getting Started with Zombie.js [Article] DWR Java AJAX User Interface: Basic Elements (Part 1) [Article]
Read more
  • 0
  • 0
  • 8383

article-image-angularjs-0
Packt
20 Aug 2014
15 min read
Save for later

AngularJS

Packt
20 Aug 2014
15 min read
In this article, by Rodrigo Branas, author of the book, AngularJS Essentials, we will go through the basics of AngularJS. Created by Miško Hevery and Adam Abrons in 2009, AngularJS is an open source, client-side JavaScript framework that promotes a high productivity web development experience. It was built over the belief that declarative programming is the best choice to construct the user's interface, while imperative programming is much better and preferred to implement the application's business logic. To achieve that, AngularJS empowers the traditional HTML by extending its current vocabulary, making the life of developers easier. The result is the development of expressive, reusable, and maintainable application components, leaving behind a lot of unnecessary code and keeping the team focused on the valuable and important things. (For more resources related to this topic, see here.) Architectural concepts It's been a long time since the famous Model-View-Controller, also known as MVC, started to be widely used in the software development industry, thereby becoming one of the legends of the enterprise architecture design. Basically, the model represents the knowledge that the view is responsible to present, while the controller mediates their relationship. However, these concepts are a little bit abstract, and this pattern may have different implementations depending on the language, platform, and purposes of the application. After a lot of discussions about which architectural pattern the framework follows, its authors declared that from now on, AngularJS is adopting Model-View-Whatever (MVW). Regardless of the name, the most important benefit is that the framework provides a clear separation of the concerns between the application layers, providing modularity, flexibility, and testability. In terms of concepts, a typical AngularJS application consists primarily of view, model, and controller, but there are other important components, such as services, directives, and filters. The view, also called template, is entirely written in HTML, which becomes a great opportunity to see web designers and JavaScript developers working side-by-side. It also takes advantage of the directives mechanism, a kind of extension of the HTML vocabulary that brings the ability to perform the programming language tasks, such as iterating over an array or even evaluating an expression conditionally. Behind the view, there is the controller. At first, the controller contains all business logic implementation used by the view. However, as the application grows, it becomes really important to perform some refactoring activities, such as moving the code from the controller to other components like services, in order to keep the cohesion high. The connection between the view and the controller is done by a shared object called scope. It is located between them and is used to exchange information related to the model. The model is a simple Plain-Old-JavaScript-Object (POJO). It looks very clear and easy to understand, bringing simplicity to the development by not requiring any special syntax to be created. Setting up the framework The configuration process is very simple and in order to set up the framework, we start by importing the angular.js script to our HTML file. After that, we need to create the application module by calling the module function, from the Angular's API, with it's name and dependencies. With the module already created, we just need to place the ng-app attribute with the module's name inside the html element or any other that surrounds the application. This attribute is important because it supports the initialization process of the framework. In the following code, there is an introductory application about a parking lot. At first, we are able to add and also list the parked cars, storing it’s plate in memory. Throughout the book, we will evolve this parking control application by incorporating each newly studied concept. index.html <!doctype html> <!-- Declaring the ng-app --> <html ng-app="parking"> <head> <title>Parking</title> <!-- Importing the angular.js script --> <script src="angular.js"></script> <script> // Creating the module called parking var parking = angular.module("parking", []); // Registering the parkingCtrl to the parking module parking.controller("parkingCtrl", function ($scope) { // Binding the car’s array to the scope $scope.cars = [ {plate: '6MBV006'}, {plate: '5BBM299'}, {plate: '5AOJ230'} ]; // Binding the park function to the scope $scope.park = function (car) { $scope.cars.push(angular.copy(car)); delete $scope.car; }; }); </script> </head> <!-- Attaching the view to the parkingCtrl --> <body ng-controller="parkingCtrl"> <h3>[Packt] Parking</h3> <table> <thead> <tr> <th>Plate</th> </tr> </thead> <tbody> <!-- Iterating over the cars --> <tr ng-repeat="car in cars"> <!-- Showing the car’s plate --> <td>{{car.plate}}</td> </tr> </tbody> </table> <!-- Binding the car object, with plate, to the scope --> <input type="text" ng-model="car.plate"/> <!-- Binding the park function to the click event --> <button ng-click="park(car)">Park</button> </body> </html> The ngController, was used to bind the parkingCtrl to the view while the ngRepeat iterated over the car's array. Also, we employed expressions like {{car.plate}} to display the plate of the car. Finally, to add new cars, we applied the ngModel, which creates a new object called car with the plate property, passing it as a parameter of the park function, called through the ngClick directive. To improve the loading page performance, it is recommended to use the minified and obfuscated version of the script that can be identified by angular.min.js. Both minified and regular distributions of the framework can be found on the official site of AngularJS, that is, http://www.angularjs.org, or they can be directly referenced to Google Content Delivery Network (CDN). What is a directive? A directive is an extension of the HTML vocabulary that allows the creation of new behaviors. This technology lets the developers create reusable components that can be used within the whole application and even provide their own custom components. It may be applied as an attribute, element, class, and even as a comment, by using the camelCase syntax. However, because HTML is case-insensitive, we need to use a lowercase form. For the ngModel directive, we can use ng-model, ng:model, ng_model, data-ng-model, and x-ng-model in the HTML markup. Using AngularJS built-in directives By default, the framework brings a basic set of directives, such as iterate over an array, execute a custom behavior when an element is clicked, or even show a given element based on a conditional expression and many others. ngBind This directive is generally applied to a span element and replaces the content of the element with the result of the provided expression. It has the same meaning as that of the double curly markup, for example, {{expression}}. Why would anyone like to use this directive when a less verbose alternative is available? This is because when the page is being compiled, there is a moment when the raw state of the expressions is shown. Since the directive is defined by the attribute of the element, it is invisible to the user. Here is an example of the ngBind directive usage: index.html <!doctype html> <html ng-app="parking"> <head> <title>[Packt] Parking</title> <script src="angular.js"></script> <script> var parking = angular.module("parking", []); parking.controller("parkingCtrl", function ($scope) { $scope.appTitle = "[Packt] Parking"; }); </script> </head> <body ng-controller="parkingCtrl"> <h3 ng-bind="appTitle"></h3> </body> </html> ngRepeat The ngRepeat directive is really useful to iterate over arrays and objects. It can be used with any kind of element such as rows of a table, elements of a list, and even options of select. We must provide a special repeat expression that describes the array to iterate over and the variable that will hold each item in the iteration. The most basic expression format allows us to iterate over an array, attributing each element to a variable: variable in array In the following code, we will iterate over the cars array and assign each element to the car variable: index.html <!doctype html> <html ng-app="parking"> <head> <title>[Packt] Parking</title> <script src="angular.js"></script> <script> var parking = angular.module("parking", []); parking.controller("parkingCtrl", function ($scope) { $scope.appTitle = "[Packt] Parking"; $scope.cars = []; }); </script> </head> <body ng-controller="parkingCtrl"> <h3 ng-bind="appTitle"></h3> <table> <thead> <tr> <th>Plate</th> <th>Entrance</th> </tr> </thead> <tbody> <tr ng-repeat="car in cars"> <td><span ng-bind="car.plate"></span></td> <td><span ng-bind="car.entrance"></span></td> </tr> </tbody> </table> </body> </html> ngModel The ngModel directive attaches the element to a property in the scope, binding the view to the model. In this case, the element could be input (all types), select, or textarea. <input type="text" ng-model="car.plate" placeholder="What's the plate?" /> There is an important advice regarding the use of this directive. We must pay attention to the purpose of the field that is using the ngModel directive. Every time the field is being part of the construction of an object, we must declare in which object the property should be attached. In this case, the object that is being constructed is a car, so we use car.plate inside the directive expression. However, sometimes it might occur that there is an input field that is just used to change a flag, allowing the control of the state of a dialog or another UI component. In these cases, we may use the ngModel directive without any object, as far as it will not be used together with other properties or even persisted. ngClick and other event directives The ngClick directive is one of the most useful kinds of directives in the framework. It allows you to bind any custom behavior to the click event of the element. The following code is an example of the usage of the ngClick directive calling a function: index.html <!doctype html> <html ng-app="parking"> <head> <title>[Packt] Parking</title> <script src="angular.js"></script> <script> var parking = angular.module("parking", []); parking.controller("parkingCtrl", function ($scope) { $scope.appTitle = "[Packt] Parking"; $scope.cars = []; $scope.park = function (car) { car.entrance = new Date(); $scope.cars.push(car); delete $scope.car; }; }); </script> </head> <body ng-controller="parkingCtrl"> <h3 ng-bind="appTitle"></h3> <table> <thead> <tr> <th>Plate</th> <th>Entrance</th> </tr> </thead> <tbody> <tr ng-repeat="car in cars"> <td><span ng-bind="car.plate"></span></td> <td><span ng-bind="car.entrance"></span></td> </tr> </tbody> </table> <input type="text" ng-model="car.plate" placeholder="What's the plate?" /> <button ng-click="park(car)">Park</button> </body> </html> Here there is another pitfall. Inside the ngClick directive, we call the park function, passing the car as a parameter. As far as we have access to the scope through the controller, would not be easier if we just access it directly, without passing any parameter at all? Keep in mind that we must take care of the coupling level between the view and the controller. One way to keep it low is by avoid reading the scope object directly from the controller, replacing this intention by passing everything it need by parameter from the view. It will increase the controller testability and also make the things more clear and explicitly. Other directives that have the same behavior, but are triggered by other events, are ngBlur, ngChange, ngCopy, ngCut, ngDblClick, ngFocus, ngKeyPress, ngKeyDown, ngKeyUp, ngMousedown, ngMouseenter, ngMouseleave, ngMousemove, ngMouseover, ngMouseup, and ngPaste. Filters The filters are, associated with other technologies like directives and expressions, responsible for the extraordinary expressiveness of framework. It lets us easily manipulate and transform any value, not only combined with expressions inside a template, but also injected in other components like controllers and services. It is really useful when we need to format date and money according to our current locale or even support the filtering feature of a grid component. Filters are the perfect answer to easily perform any data manipulating. currency The currency filter is used to format a number based on a currency. The basic usage of this filter is without any parameter: {{ 10 | currency}} The result of the evaluation will be the number $10.00, formatted and prefixed with the dollar sign. In order to achieve the correct output, in this case R$10,00 instead of R$10.00, we need to configure the Brazilian (PT-BR) locale, available inside the AngularJS distribution package. There, we may find locales to the most part of the countries and we just need to import it to our application such as: <script src="js/lib/angular-locale_pt-br.js"></script> After import the locale, we will not need to use the currency symbol anymore because it's already wrapped inside. Besides the currency, the locale also defines the configuration of many other variables like the days of the week and months, very useful when combined with the next filter used to format dates. date The date filter is one of the most useful filters of the framework. Generally, a date value comes from the database or any other source in a raw and generic format. In this way, filters like that are essential to any kind of application. Basically, we can use this filter by declaring it inside any expression. In the following example, we use the filter on a date variable attached to the scope. {{ car.entrance | date }} The output will be Dec 10, 2013. However, there are thousands of combinations that we can make with the optional format mask. {{ car.entrance | date:'MMMM dd/MM/yyyy HH:mm:ss' }} Using this format, the output changes to December 10/12/2013 21:42:10. filter Have you ever needed to filter a list of data? This filter performs exactly this task, acting over an array and applying any filtering criteria. Now, let's include in our car parking application a field to search any parked car and use this filter to do the job. index.html <input type="text" ng-model="criteria" placeholder="What are you looking for?" /> <table> <thead> <tr> <th></th> <th>Plate</th> <th>Color</th> <th>Entrance</th> </tr> </thead> <tbody> <tr ng-class="{selected: car.selected}" ng-repeat="car in cars | filter:criteria" > <td> <input type="checkbox" ng-model="car.selected" /> </td> <td>{{car.plate}}</td> <td>{{car.color}}</td> <td>{{car.entrance | date:'dd/MM/yyyy hh:mm'}}</td> </tr> </tbody> </table> The result is really impressive. With an input field and the filter declaration we did the job. Integrating the backend with AJAX AJAX, also known as Asynchronous JavaScript and XML, is a technology that allows the applications to send and retrieve data from the server asynchronously, without refreshing the page. The $http service wraps the low-level interaction with the XMLHttpRequest object, providing an easy way to perform calls. This service could be called by just passing a configuration object, used to set many important information like the method, the URL of the requested resource, the data to be sent, and many others: $http({method: "GET", url: "/resource"}) .success(function (data, status, headers, config, statusText) { }) .error(function (data, status, headers, config, statusText) { }); To make it easier to use, there are the following shortcuts methods available for this service. In this case, the configuration object is optional. $http.get(url, [config]) $http.post(url, data, [config]) $http.put(url, data, [config]) $http.head(url, [config]) $http.delete(url, [config]) $http.jsonp(url, [config]) Now, it’s time to integrate our parking application with the back-end by calling the resource cars with the method GET. It will retrieve the cars, binding it to the $scope object. In case of something went wrong, we are going to log it to the console. controllers.js parking.controller("parkingCtrl", function ($scope, $http) { $scope.appTitle = "[Packt] Parking"; $scope.park = function (car) { car.entrance = new Date(); $scope.cars.push(car); delete $scope.car; }; var retrieveCars = function () { $http.get("/cars") .success(function(data, status, headers, config) { $scope.cars = data; }) .error(function(data, status, headers, config) { switch(status) { case 401: { $scope.message = "You must be authenticated!" break; } case 500: { $scope.message = "Something went wrong!"; break; } } console.log(data, status); }); }; retrieveCars(); }); Summary This article introduced you to the fundamentals of AngularJS in order to design and construct reusable, maintainable, and modular web applications. Resources for Article: Further resources on this subject: AngularJS Project [article] Working with Live Data and AngularJS [article] CreateJS – Performing Animation and Transforming Function [article]
Read more
  • 0
  • 0
  • 8380
article-image-setting-tools-build-applications-using-jbpm-part-1
Packt
18 Jan 2010
15 min read
Save for later

Setting Up Tools to Build Applications Using jBPM: Part 1

Packt
18 Jan 2010
15 min read
Background about the jBPM project In this section, we will talk about where the jBPM framework is located inside the JBoss projects. As we know, JBoss jBPM was created and maintained for JBoss. JBoss is in charge of developing middleware "enterprise" software in Java. It is middleware because it is a type of software to make or run software, and "enterprise", as it is focused on big scenarios. This enterprise does not necessarily mean Java EE. It is also interesting to know that JBoss was bought from a company called Red Hat (famous for the Linux distribution with the same name, and also in charge of the Fedora community distribution). In order to get the right first impression about the framework, you will need to know a little about other products that JBoss has developed and where this framework is located and focused inside the company projects. At this moment, the only entry point that we have is the JBoss community page, http://www.jboss.org/. This page contains the information about all the middleware projects that JBoss is developing (all open source). If we click on the Projects link in the top menu, we are going to be redirected to a page that shows us the following image: This image shows us one important major central block for the JBoss Application Server, which contains a lot of projects intended to run inside this application server. The most representative modules are: JBoss Web: The web container based on Tomcat Web Server JBoss EJB3: EJB3 container that is standard EJB3 compliance for Java EE 5 Hibernate: The world-renowned Object Relational Mapping (ORM) framework Seam: The new web framework to build rich Internet applications JBoss Messaging: The default JMS provider that enables high performance, scalable, clustered messaging for Java On top of that, we can see two frameworks for Web Interface design (RichFaces/Ajax4jsf and Gravel) based on the components, which can be used in any web application that you code. And then, on top of it all, we can see three important blocks—Portal, Integration, and Telecom. As you can imagine, we are focused on the Integration block that contains three projects inside it. As you can see, this Integration block is also outside the JBoss Application Server boundaries. Therefore, we might suppose that these three products will run without any dependency from JBoss or any other application server. Now we are going to talk about these three frameworks, which have different focuses inside the integration field. JBoss Drools Drools is, of late, focused on business knowledge, and because it was born as an inference engine, it will be in charge of using all that business knowledge in order to take business actions based on this knowledge for a specific situation. You can find out more information about this framework (now redefined as Business Logic integration Platform) at http://www.drools.org. JBoss ESB It is a product focused on supplying an Enterprise Service Bus (ESB), which allows us to use different connectors to communicate with heterogeneous systems that were created in different languages. These use different protocols for communication. You can find out more information about this project at http://www.jboss.org/jbossesb/. JBoss jBPM jBPM has a process-centric philosophy. This involves all the APIs and tools that are related to the processes and how to manage them. The framework perspective is always centered on the business process that we describe. Also, the services available inside the framework are only for manipulating the processes. All the other things that we want or need for integration with our processes will be delegated to third-party frameworks or tools. Now, if we enter into the official page of jBPM (http://www.jbpm.org), we are going to see all the official information and updates about the framework. It is important to notice the home page, which shows us the following image: This is the first image that developers see when they get interested in jBPM. This image shows us the component distribution inside the jBPM framework project. Understanding these building blocks (components) will help us to understand the code of the framework and each part's functionality. Most of the time, this image is not clearly understood, so let's analyze it! Supported languages One of the important things that the image shows is the multi-language support for modeling processes in different scenarios. We can see that three languages are currently supported/proposed by the framework with the possibility to plug in new languages that we need, in order to represent our business processes with extra technology requirements. These supported languages are selected according to our business scenario and the technology that this scenario requires. The most general and commonly used language is jBPM Process Definition Language (jPDL). This language can be used in  situations where we are defining the project architecture and the technology that the project will use. In most of the cases, jPDL will be the correct choice, because it brings the flexibility to model any kind of situation, the extensibility to expand our process language with new words to add extra functionality to the base implementation, and no technology pluggability limitation, thereby allowing us to interact with any kind of external services and systems. That is why jPDL can be used in almost all situations. If you don't have any technology restriction in your requirements, this language is recommended. jBPM also implements the Business Process Execution Language (BPEL), which is broadly used to orchestrate Web Services classes between different systems. To support business scenarios where all the interactions are between web services, I recommend that you make use of this language, only if you are restricted to using a standard like BPEL, in order to model your business process. PageFlow is the last one shown in the image. This language will be used when you use the JBoss Seam framework and want to describe how your web pages are synchronized to fulfill some requirements. These kind of flows are commonly used to describe navigation flow possibilities that a user will have in a website. Web applications will benefit enormously from this, because the flow of the web application will be decoupled from the web application code, letting us introduce changes without modifying the web pages themselves. At last, the language pluggability feature is represented with the ellipse (...). This will be required in situations wherein the available languages are not enough to represent our business scenarios. This could happen when a new standard like BPEL or  BPMN arises, or if our company has its own language to represent business processes. In these kind of situations, we will need to implement our custom language on top of the process' virtual machine. This is not an easy task and it is important for you to know that it is not a trivial thing to implement an entire language. So, here we will be focused on learning jPDL in depth, to understand all of its features and how to extend it in order to fulfill our requirements. Remember that jPDL is a generic language that allows us to express almost every situation. In other words, the only situation where jPDL doesn't fit is where the process definition syntax doesn't allow us to represent our business process or where the syntax needs to follow a standard format like BPMN or BPEL. Also, it is important to notice that all these languages are separate from the Process Virtual Machine (PVM), the block on the bottom-left of the image, which will execute our defined process. PVM is like the core of the framework and understands all the languages that are defined. This virtual machine will know how to execute them and how to behave for each activity in different business scenarios. When we begin to understand the jPDL language in depth, we will see how PVM behaves for each activity described in our process definitions. Other modules Besides the PVM and all the languages, we can also see some other modules that implement extra functionality, which will help us with different requirements. The following list contains a brief description of each module: Graphical Process Designer (GPD) module: It is the graphical process designer module implemented as an Eclipse plugin. Identity module: This module is a proof of concept, out-of-the-box working module used to integrate business roles for our processes. This module is focused on letting us represent people/users inside the process definition and execution. This module shows us a simple structure for users and groups that can be used inside our processes. For real scenarios, this module will help us to understand how we will map our users' structures with the jBPM framework. Task ManaGeMenT (TaskMGMT) module: This module's functionality involves dealing with all the integration that the people/employees/business roles have with the processes. This module will help us to manage all the necessary data to create application clients, which the business roles will use in their everyday work. Enterprise module: This module brings us extra functionality for enterprise environments. Now that we know how the components are distributed inside the framework, we can jump to the jPDL section of jBPM's official web page. Here we will find the third image that all the developers will see when they get started with jBPM. Let's analyze this image to understand why and how the framework can be used in different platforms. This image tries to give us an example of how jBPM could be deployed on a web server or an application server. Please, keep in mind that this is not the only way that jBPM could be deployed on, or embedded in, an application, because jBPM can also be used in a standalone application. In addition, this image shows us some of the BPM stages that are implemented. For example, we can see how the designed processes will be formalized in the jPDL XML syntax in Graphical Process Designer (GPD)— here called the Eclipse jPDL Editor. On the other side of the image, we can see the execution stage implemented inside a container that could be an Enterprise Container (such as JBoss Application Server) or just a web server (such as Tomcat or Jetty). This distinction is made with the extensions of the deployed files (war, for Web Archives, and ear, for Enterprise Archives). In this container, it is important to note the jpdl-jbpm.jar archive that contains the PVM and the language definition, which lets us understand the process defined in jPDL. Also, we have the jbpm-identity.jar as a result of the Identity Module that we have seen in the other image. Besides, we have the hibernate.jar dependency. This fact is very important to note, because our processes will be persisted with Hibernate and we need to know how to adapt this to our needs. The last thing that we need to see is the Firefox/Internet Explorer logo on top of the image, which tries to show us how our clients (users), all the people who interact and make activities in our processes will talk (communicate) with the framework. Once again, HTTP interaction is not the only way to interact with the processes, we can implement any kind of interactions (such as JMS for enterprise messaging, Web Services to communicate with heterogeneous systems, mails for some kind of flexibility, SMS, and so on). Here we get a first impression about the framework, now we are ready to go ahead and install all the tools that we need, in order to start building applications. Tools and software For common tools such as Java Development Kit, IDE installation, database installation, and so on, only the key points will be discussed. In jBPM tooling, a detailed explanation will follow the download and installation process. We will be going into the structure detail and specification in depth; about how and why we are doing this installation. If you are an experienced developer, you can skip this section and go directly to the jBPM installation section. In order to go to the jBPM installation section straightaway, you will need to have the following software installed correctly: Java Development Kit 1.5 or higher (This is the first thing that Java developers learn. If you don't know how to install it, please take a look at the following link: http://java.sun.com/javase/6/webnotes/install/index.html.) Maven 2.0.9 or higher A Hibernate supported database, here we will use MySQL You will need to have downloaded the Java Connector for your selected database JBoss 5.0.1.GA installed (If you are thinking about creating Enterprise Applications, you will need JBoss AS installed. If you only want to create web applications with Tomcat or Jetty installed, this will be fine.) Eclipse IDE 3.4 Ganymede (Eclipse IDE 3.4 Ganymede is the suggested version. You can try it with other versions, but this is the one tested in the article.) An SVN client, here we will use Tortoise SVN (Available for Windows only, you can also use a subversion plugin for Eclipse or for your favorite IDE.) If you have all this software up and running, you can jump to the next section. If not, here we will see a brief introduction of each one of them with some reasons that explain why we need each of these tools. Maven—why do I need it? Maven is an Apache project that helps us to build, maintain, and manage our Java Application projects. One of the main ideas behind Maven is to solve all the dependency problems between our applications and all the framework libraries that we use. If you read the What is Maven? page (http://maven.apache.org/what-is-maven.html), you will find the key point behind this project. The important things that we will use here and in your diary work will be: A standard structure for all your projects Centralized project and dependencies description Standard structure for all your projects Maven proposes a set of standard structures to build our Java projects. The project descriptor that we need to write/create depends on the Java project type that we want to build. The main idea behind it is to minimize the configuration files to build our applications. A standard is proposed to build each type of application. You can see all the suggested standard structure on the official Maven page: http://maven.apache.org/guides/introduction/introduction-to-thestandard-directory-layout.html. Centralized project and dependencies description When we are using Maven, our way of building applications and managing the dependencies needed by these applications changes a lot. In Maven, the concept of Project Object Model (POM) is introduced. This POM will define our project structure, dependencies, and outcome(s) in XML syntax. This means that we will have just one file where we will define the type of project we are building, the first order dependencies that the project will have, and the kind of outcome(s) that we are expecting after we build our project. Take a look at the following pom.xml file: <?xml version="1.0" encoding="UTF-8"?><project xsi_schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.jbpm.examples</groupId> <artifactId>chapter02.homeworkSolution</artifactId> <packaging>jar</packaging> <version>1.0-SNAPSHOT</version> <name>chapter02.homeworkSolution</name> <url>http://maven.apache.org</url> <build> <plugins> <plugin> <artifactId>maven-compiler-plugin</artifactId> <version>2.0.2</version> <configuration> <source>1.5</source> <target>1.5</target> </configuration> </plugin> </plugins> </build><dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>3.8.1</version> <scope>test</scope> </dependency></dependencies></project> We are basically defining all the mentioned characteristics of our project. All this information is deduced from the packaging attribute, which in this case is: <packaging>jar</packaging> The standard structure of directories will be used in order to know where the source code is located and where the compiled outcome will be placed. Maven installation Getting maven installed is a very simple task. You should download the Maven binaries from the official page(http://maven.apache.org). This will be a .zip file, or a .tar.gz file, which you will only need to uncompress in the programs directory. You will also add the bin directory to the system Path variable. With that, you will be able to call the mvn command from the console. To test whether Maven is working properly, you can open the Windows console and type mvn. You should get something like this: This output shows us that Maven is correctly installed. However, as it is installed in C:Documents and Settingssalaboy21 (the installation directory) where there is no project descriptor, the build failed. I strongly recommend that you read and understand the Getting Started section in the official Maven documentation at http://maven.apache.org/guides/getting-started/index.html.
Read more
  • 0
  • 0
  • 8371

article-image-c-compiler-device-drivers-and-useful-developing-techniques
Packt
17 Mar 2017
22 min read
Save for later

C compiler, Device Drivers and Useful Developing Techniques

Packt
17 Mar 2017
22 min read
In this article by Rodolfo Giometti, author of the book GNU/Linux Rapid Embedded Programming, in this article we’re going to focusing our attention to the C compiler (with its counter part: the cross-compiler) and when we have (or we can choose to) to use the native or cross-compilation and the differences between them. (For more resources related to this topic, see here.) Then we’ll see some kernel stuff used later in this article (configuration, recompilation and the device tree) and then we’ll look a bit deeper at the device drivers, how they can be compiled and how they can be put into a kernel module (that is kernel code that can be loaded at runtime). We'll present different kinds of computer peripherals and, for each of them, we'll try to explain how the corresponding device driver works starting from the compilation stage through the configuration till the final usage. As example we’ll try to implement a very simple driver in order to give to the reader some interesting points of view and very simple advices about kernel programming (which is not covered by this article!). We’re going to present the root filesystem’s internals and we’ll spend some words about a particular root filesystem that can be very useful during the early developing stages: the Network File System. As final step we’ll propose the usage of an emulator in order to execute a complete target machine’s Debian distribution on a host PC. This article still is part of the introductory part of this article, experienced developers whose already well know these topics may skip this article but the author's suggestion still remains the same, that is to read the article anyway in order to discover which developing tools will be used in the article and, maybe, some new technique to manage their programs. The C compiler The C compiler is a program that translate the C language) into a binary format that the CPU can understand and execute. This is the vary basic way (and the most powerful one) to develop programs into a GNU/Linux system. Despite this fact most developer prefer using another high level languages rather than C due the fact the C language has no garbage collection, has not objects oriented programming and other issue, giving up part of the execution speed that a C program offers, but if we have to recompile the kernel (the Linux kernel is written in C – plus few assembler), to develop a device driver or to write high performance applications then the C language is a must-have. We can have a compiler and a cross-compiler and till now, we’ve already used the cross-compiler several times to re-compile the kernel and the bootloaders, however we can decide to use a native compiler too. In fact using native compilation may be easier but, in most cases, very time consuming that’s why it’s really important knowing the pros and cons. Programs for embedded systems are traditionally written and compiled using a cross-compiler for that architecture on a host PC. That is we use a compiler that can generate code for a foreign machine architecture, meaning a different CPU instruction set from the compiler host's one. Native & foreign machine architecture For example the developer kits shown in this article are an ARM machines while (most probably) our host machine is an x86 (that is a normal PC), so if we try to compile a C program on our host machine the generated code cannot be used on an ARM machine and vice versa. Let's verify it! Here the classic Hello World program below: #include <stdio.h> int main() { printf("Hello Worldn"); return 0; } Now we compile it on my host machine using the following command: $ make CFLAGS="-Wall -O2" helloworld cc -Wall -O2 helloworld.c -o helloworld Careful reader should notice here that we’ve used command make instead of the usual cc. This is a perfectly equivalent way to execute the compiler due the fact, even if without a Makefile, command make already knows how to compile a C program. We can verify that this file is for the x86 (that is the PC) platform by using the file command: $ file helloworld helloworld: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0f0db5e65e1cd09957ad06a7c1b7771d949dfc84, not stripped Note that the output may vary according to the reader's host machine platform. Now we can just copy the program into one developer kit (for instance the the BeagleBone Black) and try to execute it: root@bbb:~# ./helloworld -bash: ./helloworld: cannot execute binary file As we expected the system refuses to execute code generated for a different architecture! On the other hand, if we use a cross-compiler for this specific CPU architecture the program will run as a charm! Let's verify this by recompiling the code but paying attention to specify that we wish to use the cross-compiler instead. So delete the previously generated x86 executable file (just in case) by using the rm helloworld command and then recompile it using the cross-compiler: $ make CC=arm-linux-gnueabihf-gcc CFLAGS="-Wall -O2" helloworld arm-linux-gnueabihf-gcc -Wall -O2 helloworld.c -o helloworld Note that the cross-compiler's filename has a special meaning: the form is <architecture>-<platform>-<binary-format>-<tool-name>. So the filename arm-linux-gnueabihf-gcc means: ARM architecture, Linux platform, gnueabihf (GNU EABI Hard-Float) binary format and gcc (GNU C Compiler) tool. Now we use the file command again to see if the code is indeed generated for the ARM architecture: $ file helloworld helloworld: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=31251570b8a17803b0e0db01fb394a6394de8d2d, not stripped Now if we transfer the file as before on the BeagleBone Black and try to execute it, we get: root@bbb:~# ./helloworld Hello World! Therefore we see the cross-compiler ensures that the generated code is compatible with the architecture we are executing it on. In reality in order to have a perfectly functional binary image we have to make sure that the library versions, header files (also the headers related to the kernel) and cross compiler options match the target exactly or, at least, they are compatible. In fact we cannot execute cross-compiled code against the glibc on a system having, for example, musl libc (or it can run in a no predictable manner). In this case we have perfectly compatible libraries and compilers but, in general, the embedded developer should perfectly know what he/she is doing. A common trick to avoid compatibility problems is to use static compilation but, in this case, we get huge binary files. Now the question is: when should we use the compiler and when the cross-compiler? We should compile on an embedded system because: We can (see below why). There would be no compatibility issues as all the target libraries will be available. In cross-compilation it becomes hell when we need all the libraries (if the project uses any) in the ARM format on the host PC. So we not only have to cross-compile the program but also its dependencies. And if the same version dependencies are not installed on the embedded system's rootfs, then good luck with troubleshooting! It's easy and quick. We should cross-compile because: We are working on a large codebase and we don't want to waste too much time compiling the program on the target, which may take from several minutes to several hours (or even it may result impossible). This reason might be strong enough to overpower the other reasons in favor of compiling on the embedded system itself. PCs nowadays have multiple cores so the compiler can process more files simultaneously. We are building a full Linux system from scratch. In any case, below, we will show an example of both native compilation and cross-compilation of a software package, so the reader may well understand the differences between them. Compiling a C program As first step let's see how we can compile a C program. To keep it simple we’ll start compiling a user-space program them in the next sections, we’re going to compile some kernel space code. Knowing how to compile an C program can be useful because it may happen that a specific tool (most probably) written in C is missing into our distribution or it’s present but with an outdated version. In both cases we need to recompile it! To show the differences between a native compilation and a cross-compilation we will explain both methods. However a word of caution for the reader here, this guide is not exhaustive at all! In fact the cross-compilation steps may vary according to the software packages we are going to cross-compile. The package we are going to use is the PicoC interpreter. Each Real-Programmers(TM) know the C compiler, which is normally used to translate a C program into the machine language, but (maybe) not all of them know that a C interpreter exists too! Actually there are many C interpreters, but we focus our attention on PicoC due its simplicity in cross-compiling it. As we already know, an interpreter is a program that converts the source code into executable code on the fly and does not need to parse the complete file and generate code at once. This is quite useful when we need a flexible way to write brief programs to resolve easy tasks. In fact to fix bugs in the code and/or changing the program behavior we simply have to change the program source and then re-executing it without any compilation at all. We just need an editor to change our code! For instance, if we wish to read some bytes from a file we can do it by using a standard C program, but for this easy task we can write a script for an interpreter too. Which interpreter to choose is up to developer and, since we are C programmers, the choice is quite obvious. That's why we have decided to use PicoC. Note that the PicoC tool is quite far from being able to interpret all C programs! In fact this tool implements a fraction of the features of a standard C compiler; however it can be used for several common and easy tasks. Please, consider the PicoC as an education tool and avoid using it in a production environment! The native compilation Well, as a first step we need to download the PicoC source code from its repository at: http://github.com/zsaleeba/picoc.git into our embedded system. This time we decided to use the BeagleBone Black and the command is as follows: root@bbb:~# git clone http://github.com/zsaleeba/picoc.git When finished we can start compiling the PicoC source code by using: root@bbb:~# cd picoc/ root@bbb:~/picoc# make Note that if we get the error below during the compilation we can safely ignore it: /bin/sh: 1: svnversion: not found However during the compilation we get: platform/platform_unix.c:5:31: fatal error: readline/readline.h: No such file or directory #include <readline/readline.h> ^ compilation terminated. <builtin>: recipe for target 'platform/platform_unix.o' failed make: *** [platform/platform_unix.o] Error 1 Bad news, we have got an error! This because the readline library is missing; hence we need to install it to keep this going. In order to discover which package's name holds a specific tool, we can use the following command to discover the package that holds the readline library: root@bbb:~# apt-cache search readline The command output is quite long, but if we carefully look at it we can see the following lines: libreadline5 - GNU readline and history libraries, run-time libraries libreadline5-dbg - GNU readline and history libraries, debugging libraries libreadline-dev - GNU readline and history libraries, development files libreadline6 - GNU readline and history libraries, run-time libraries libreadline6-dbg - GNU readline and history libraries, debugging libraries libreadline6-dev - GNU readline and history libraries, development files This is exactly what we need to know! The required package is named libreadline-dev. In the Debian distribution all libraries packages are prefixed by the lib string while the -dev postfix is used to mark the development version of a library package. Note also that we choose the package libreadline-dev intentionally leaving the system to choose to install version 5 o 6 of the library. The development version of a library package holds all needed files whose allow the developer to compile his/her software to the library itself and/or some documentation about the library functions. For instance, into the development version of the readline library package (that is into the package libreadline6-dev) we can find the header and the object files needed by the compiler. We can see these files using the following command: #root@bbb:~# dpkg -L libreadline6-dev | egrep '.(so|h)' /usr/include/readline/rltypedefs.h /usr/include/readline/readline.h /usr/include/readline/history.h /usr/include/readline/keymaps.h /usr/include/readline/rlconf.h /usr/include/readline/tilde.h /usr/include/readline/rlstdc.h /usr/include/readline/chardefs.h /usr/lib/arm-linux-gnueabihf/libreadline.so /usr/lib/arm-linux-gnueabihf/libhistory.so So let's install it: root@bbb:~# aptitude install libreadline-dev When finished we can relaunch the make command to definitely compile our new C interpreter: root@bbb:~/picoc# make gcc -Wall -pedantic -g -DUNIX_HOST -DVER="`svnversion -n`" -c -o clibrary.o clibrary.c ... gcc -Wall -pedantic -g -DUNIX_HOST -DVER="`svnversion -n`" -o picoc picoc.o table.o lex.o parse.o expression.o heap.o type.o variable.o clibrary.o platform.o include.o debug.o platform/platform_unix.o platform/library_unix.o cstdlib/stdio.o cstdlib/math.o cstdlib/string.o cstdlib/stdlib.o cstdlib/time.o cstdlib/errno.o cstdlib/ctype.o cstdlib/stdbool.o cstdlib/unistd.o -lm -lreadline Well now the tool is successfully compiled as expected! To test it we can use again the standard Hello World program above but with a little modification, in fact the main() function is not defined as before! This is due the fact PicoC returns an error if we use the typical function definition. Here the code: #include <stdio.h> int main() { printf("Hello Worldn"); return 0; } Now we can directly execute it (that is without compiling it) by using our new C interpreter: root@bbb:~/picoc# ./picoc helloworld.c Hello World An interesting feature of PicoC is that it can execute C source file like a script, that is we don't need to specify a main() function as C requires and the instructions are executed one by one from the beginning of the file as a normal scripting language does. Just to show it we can use the following script which implements the Hello World program as C-like script (note that the main() function is not defined!): printf("Hello World!n"); return 0; If we put the above code into the file helloworld.picoc we can execute it by using: root@bbb:~/picoc# ./picoc -s helloworld.picoc Hello World! Note that this time we add the -s option argument to the command line in order to instruct the PicoC interpreter that we wish using its scripting behavior. The cross-compilation Now let's try to cross-compile the PicoC interpreter on the host system. However, before continuing, we’ve to point out that this is just an example of a possible cross-compilation useful to expose a quick and dirty way to recompile a program when the native compilation is not possible. As already reported above the cross-compilation works perfectly for the bootloader and the kernel while for user-space application we must ensure that all involved libraries (and header files) used by the cross-compiler are perfectly compatible with the ones present on the target machine otherwise the program may not work at all! In our case everything is perfectly compatible so we can go further. As before we need to download the PicoC's source code by using the same git command as above. Then we have to enter the following command into the newly created directory picoc: $ cd picoc/ $ make CC=arm-linux-gnueabihf-gcc arm-linux-gnueabihf-gcc -Wall -pedantic -g -DUNIX_HOST -DVER="`svnversion -n`" -c -o picoc.o picoc.c ... platform/platform_unix.c:5:31: fatal error: readline/readline.h: No such file or directory compilation terminated. <builtin>: recipe for target 'platform/platform_unix.o' failed make: *** [platform/platform_unix.o] Error 1 We specify the CC=arm-linux-gnueabihf-gcc commad line option to force the cross-compilation. However, as already stated before, the cross-compilation commands may vary according to the compilation method used by the single software package. As before the system returns a linking error due to the fact that thereadline library is missing, however, this time, we cannot install it as before since we need the ARM version (specifically the armhf version) of this library and my host system is a normal PC! Actually a way to install a foreign package into a Debian/Ubuntu distribution exists, but it's not a trivial task nor it's an argument. A curious reader may take a look at the Debian/Ubuntu Multiarch at https://help.ubuntu.com/community/MultiArch. Now we have to resolve this issue and we have two possibilities: We can try to find a way to install the missing package, or We can try to find a way to continue the compilation without it. The former method is quite complex since the readline library has in turn other dependencies and we may take a lot of time trying to compile them all, so let's try to use the latter option. Knowing that the readline library is just used to implement powerful interactive tools (such as recalling a previous command line to re-edit it, etc.) and since we are not interested in the interactive usage of this interpreter, we can hope to avoid using it. So, looking carefully into the code we see that the define USE_READLINE exists and changing the code as shown below should resolve the issue allowing us to compile the tool without the readline support: $ git diff diff --git a/Makefile b/Makefile index 6e01a17..c24d09d 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,6 @@ CC=gcc CFLAGS=-Wall -pedantic -g -DUNIX_HOST -DVER="`svnversion -n`" -LIBS=-lm -lreadline +LIBS=-lm TARGET = picoc SRCS = picoc.c table.c lex.c parse.c expression.c heap.c type.c diff --git a/platform.h b/platform.h index 2d7c8eb..c0b3a9a 100644 --- a/platform.h +++ b/platform.h @@ -49,7 +49,6 @@ # ifndef NO_FP # include <math.h> # define PICOC_MATH_LIBRARY -# define USE_READLINE # undef BIG_ENDIAN # if defined(__powerpc__) || defined(__hppa__) || defined(__sparc__) # define BIG_ENDIAN The above output is in the unified context diff format; so the code above means that into the file Makefile the option -lreadline must be removed from variable LIBS and that into the file platform.h the define USE_READLINE must be commented out. After all the changes are in place we can try to recompile the package with the same command as before: $ make CC=arm-linux-gnueabihf-gcc arm-linux-gnueabihf-gcc -Wall -pedantic -g -DUNIX_HOST -DVER="`svnversion -n`" -c -o table.o table.c ... arm-linux-gnueabihf-gcc -Wall -pedantic -g -DUNIX_HOST -DVER="`svnversion -n`" -o picoc picoc.o table.o lex.o parse.o expression.o heap.o type.o variable.o clibrary.o platform.o include.o debug.o platform/platform_unix.o platform/library_unix.o cstdlib/stdio.o cstdlib/math.o cstdlib/string.o cstdlib/stdlib.o cstdlib/time.o cstdlib/errno.o cstdlib/ctype.o cstdlib/stdbool.o cstdlib/unistd.o -lm Great! We did it! Now, just to verify that everything is working correctly, we can simply copy the picoc file into our BeagleBone Black and test it as before. Compiling a kernel module As a special example of cross-compilation we'll take a look at a very simple code which implement a dummy module for the Linux kernel (the code does nothing but printing some messages on the console) and we’ll try to cross-compile it. Let's consider this following kernel C code of the dummy module: #include <linux/module.h> #include <linux/init.h> /* This is the function executed during the module loading */ static int dummy_module_init(void) { printk("dummy_module loaded!n"); return 0; } /* This is the function executed during the module unloading */ static void dummy_module_exit(void) { printk("dummy_module unloaded!n"); return; } module_init(dummy_module_init); module_exit(dummy_module_exit); MODULE_AUTHOR("Rodolfo Giometti <giometti@hce-engineering.com>"); MODULE_LICENSE("GPL"); MODULE_VERSION("1.0.0"); Apart some defines relative to the kernel tree the file holds two main functions  dummy_module_init() and  dummy_module_exit() and some special definitions, in particular the module_init() and module_exit(), that address the first two functions as the entry and exit functions of the current module (that is the function which are called at module loading and unloading). Then consider the following Makefile: ifndef KERNEL_DIR $(error KERNEL_DIR must be set in the command line) endif PWD := $(shell pwd) CROSS_COMPILE = arm-linux-gnueabihf- # This specifies the kernel module to be compiled obj-m += module.o # The default action all: modules # The main tasks modules clean: make -C $(KERNEL_DIR) ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- SUBDIRS=$(PWD) $@ OK, now to cross-compile the dummy module on the host PC we can use the following command: $ make KERNEL_DIR=~/A5D3/armv7_devel/KERNEL/ make -C /home/giometti/A5D3/armv7_devel/KERNEL/ SUBDIRS=/home/giometti/github/chapter_03/module modules make[1]: Entering directory '/home/giometti/A5D3/armv7_devel/KERNEL' CC [M] /home/giometti/github/chapter_03/module/dummy.o Building modules, stage 2. MODPOST 1 modules CC /home/giometti/github/chapter_03/module/dummy.mod.o LD [M] /home/giometti/github/chapter_03/module/dummy.ko make[1]: Leaving directory '/home/giometti/A5D3/armv7_devel/KERNEL' It's important to note that when a device driver is released as a separate package with a Makefile compatible with the Linux's one we can compile it natively too! However, even in this case, we need to install a kernel source tree on the target machine anyway. Not only, but the sources must also be configured in the same manner of the running kernel or the resulting driver will not work at all! In fact a kernel module will only load and run with the kernel it was compiled against. The cross-compilation result is now stored into the file dummy.ko, in fact we have: $ file dummy.ko dummy.ko: ELF 32-bit LSB relocatable, ARM, EABI5 version 1 (SYSV), BuildID[sha1]=ecfcbb04aae1a5dbc66318479ab9a33fcc2b5dc4, not stripped The kernel modules as been compiled for the SAMA5D3 Xplained but, of course, it can be cross-compiled for the other developer kits in a similar manner. So let’s copy our new module to the SAMA5D3 Xplained by using the scp command through the USB Ethernet connection: $ scp dummy.ko root@192.168.8.2: root@192.168.8.2's password: dummy.ko 100% 3228 3.2KB/s 00:00 Now, if we switch on the SAMA5D3 Xplained, we can use the modinfo command to get some information of the kernel module: root@a5d3:~# modinfo dummy.ko filename: /root/dummy.ko version: 1.0.0 license: GPL author: Rodolfo Giometti <giometti@hce-engineering.com> srcversion: 1B0D8DE7CF5182FAF437083 depends: vermagic: 4.4.6-sama5-armv7-r5 mod_unload modversions ARMv7 thumb2 p2v8 Then to load and unload it into and from the kernel we can use the insmod and rmmod commands as follow: root@a5d3:~# insmod dummy.ko [ 3151.090000] dummy_module loaded! root@a5d3:~# rmmod dummy.ko [ 3153.780000] dummy_module unloaded! As expected the dummy’s messages has been displayed on the serial console. Note that if we are using a SSH connection we have to use the dmesg or tail -f /var/log/kern.log commands to see kernel’s messages. Note also that the commands modinfo, insmod and rmmod are explained in detail in a section below. The Kernel and DTS files Main target of this article is to give several suggestions for rapid programming methods to be used on an embedded GNU/Linux system, however the main target of every embedded developer is to realize programs to manage peripherals, to monitor or to control devices and other similar tasks to interact with the real world, so we mainly need to know the techniques useful to get access to the peripheral’s data and settings. That’s why we need to know firstly how to recompile the kernel and how to configure it. Summary In this article we did a very long tour into three of the most important topics of the GNU/Linux embedded programming: the C compiler (and the cross-compiler), the kernel (and the device drivers with the device tree) and the root filesystem. Also we presented the NFS in order to have a remote root filesystem over the network and we introduced the emulator usage in order to execute foreign code on the host PC. Resources for Article: Further resources on this subject: Visualizations made easy with gnuplot [article] Revisiting Linux Network Basics [article] Fundamental SELinux Concepts [article]
Read more
  • 0
  • 0
  • 8365
Modal Close icon
Modal Close icon