How-To Tutorials

27 Jan 2011

2 min read

IBM Tuesday

27 Jan 2011

IBM Rational Team Concert 2 Essentials Improve your team productivity with Integrated Process, Planning, and Collaboration using Team Concert Enterprise Edition OUT NOW! Getting Started with IBM FileNet P8 Content Manager Install, customize, and administer the powerful FileNet enterprise content management platform OUT NOW! IBM Lotus Quickr 8.5 for Domino Administration Ensure effective and efficient team collaboration by building a solid social infrastructure with IBM Lotus Quickr 8.5 OUT NOW! IBM WebSphere Application Server v7.0 Security Secure your WebSphere applications with Java EE and JAAS security standards OUT NOW! Why not make the most of this offer and purchase a selection of IBM books, here are some popular bundles people are buying right now: + + WebSphere Application Server 7.0 Administration Guide IBM WebSphere Application Server v7.0 Security WebSphere Application Server 7.0 Administration Guide IBM WebSphere eXtreme Scale 6 + + IBM Lotus Notes 8.5 User Guide IBM Lotus Quickr 8.5 for Domino Administration IBM Lotus Sametime 8 Essentials: A User's Guide IBM Lotus Notes 8.5 User Guide + + IBM Lotus Notes and Domino 8.5.1 IBM Lotus Quickr 8.5 for Domino Administration WS-BPEL 2.0 for SOA Composite Applications with IBM WebSphere 7 Application Development for IBM WebSphere Process Server 7 and Enterprise Service Bus 7 + + IBM WebSphere Application Server v7.0 Security WS-BPEL 2.0 for SOA Composite Applications with IBM WebSphere 7 IBM Cognos 8 Planning IBM Cognos 8 Report Studio Cookbook + IBM ClearCase 7.0: Master the Tools That Monitor, Analyze, and Manage Software Configurations IBM InfoSphere Replication Server and Data Event Publisher Latest Articles On IBM:- IBM Lotus Quickr Services Overview Replication Alert Monitor: Monitoring Management Advanced Activities in BPEL FAQ on Installing and Using the IBM Lotus Sametime 8 Mobile Client IBM's BPM BlueWorks

0
0
6804

How-To Tutorials

article-image-oracle-e-business-suite-creating-bank-accounts-and-cash-forecasts

Packt

19 Aug 2011

3 min read

Oracle E-Business Suite: Creating Bank Accounts and Cash Forecasts

Packt

19 Aug 2011

3 min read

Oracle E-Business Suite 12 Financials Cookbook Take the hard work out of your daily interactions with E-Business Suite financials by using the 50+ recipes from this cookbook Introduction Oracle E-business suite The liquidity of an organization is managed in Oracle Cash Management; this includes the reconciliation of the cashbook to the bank statements, and forecasting future cash requirements. In this article, we will look at how to create bank accounts and cash forecasts. Cash management integrates with Payables, Receivables, Payroll, Treasury, and General Ledger. Let's start by looking at the cash management process: The Bank generates statements. The statements are sent to the organization electronically or by post. The Treasury Administrator loads and verifies the bank statement into cash management. The statements can also be manually entered into cash management. The loaded statements are reconciled to the cash book transactions. The results are reviewed, and amended if required. The Treasury Administrator creates the journals for transactions in the General Ledger. Creating bank accounts Oracle Cash Management provides us with the functionality to create bank accounts. In this recipe, we will create a bank account for a bank called Shepherd Bank, for one of their branches called Kings Cross branch. Getting ready Log in to Oracle E-Business Suite R12 with the username and password assigned to you by the system administrator. If you are working on the Vision demonstration database, you can use OPERATIONS/WELCOME as the USERNAME/PASSWORD. We also need to create a bank before we can create the bank account. Let's look at how to create a bank and the branch: Select the Cash Management responsibility. Navigate to Setup | Banks | Banks.(Move the mouse over the image to enlarge it.) In the Banks tab, click on the Create button. Select the Create new bank option. In the Country field, enter United States. In the Bank Name field, enter Shepherds Bank. In the Bank Number field, enter JN316. Click on the Finish button. Let's create the branch and the address: (Move the mouse over the image to enlarge it.) Click the Create Branch icon: The Country and the Bank Name are automatically entered. Click on the Continue button.(Move the mouse over the image to enlarge it.) In the Branch Name field, enter Kings Cross. Select ABA as the Branch Type. Click on the Save and Next button to create the Branch address.(Move the mouse over the image to enlarge it.) In the Branch Address form, click on the create button. In the Country field, enter United States. In the Address Line 1 field, enter 4234 Red Eagle Road. In the City field, enter Sacred Heart. In the County field, enter Renville. In the State field, enter MN. In the Postal Code field, enter 56285. Ensure that the Status field is Active. Click on the Apply button. Click on the Finish button.

0
0
6802

Packt

20 May 2016

28 min read

Visualizations Using CCC

Packt

20 May 2016

28 min read

0
0
6801

article-image-rest-apis-social-network-data-using-py2neo

Packt

14 Jul 2015

20 min read

REST APIs for social network data using py2neo

Packt

14 Jul 2015

20 min read

In this article wirtten by Sumit Gupta, author of the book Building Web Applications with Python and Neo4j we will discuss and develop RESTful APIs for performing CRUD and search operations over our social network data, using Flask-RESTful extension and py2neo extension—Object-Graph Model (OGM). Let's move forward to first quickly talk about the OGM and then develop full-fledged REST APIs over our social network data. (For more resources related to this topic, see here.) ORM for graph databases py2neo – OGM We discussed about the py2neo in Chapter 4, Getting Python and Neo4j to Talk Py2neo. In this section, we will talk about one of the py2neo extensions that provides high-level APIs for dealing with the underlying graph database as objects and its relationships. Object-Graph Mapping (http://py2neo.org/2.0/ext/ogm.html) is one of the popular extensions of py2neo and provides the mapping of Neo4j graphs in the form of objects and relationships. It provides similar functionality and features as Object Relational Model (ORM) available for relational databases py2neo.ext.ogm.Store(graph) is the base class which exposes all operations with respect to graph data models. Following are important methods of Store which we will be using in the upcoming section for mutating our social network data: Store.delete(subj): It deletes a node from the underlying graph along with its associated relationships. subj is the entity that needs to be deleted. It raises an exception in case the provided entity is not linked to the server. Store.load(cls, node): It loads the data from the database node into cls, which is the entity defined by the data model. Store.load_related(subj, rel_type, cls): It loads all the nodes related to subj of relationship as defined by rel_type into cls and then further returns the cls object. Store.load_indexed(index_name, key,value, cls): It queries the legacy index, loads all the nodes that are mapped by key-value, and returns the associated object. Store.relate(subj, rel_type, obj, properties=None): It defines the relationship between two nodes, where subj and cls are two nodes connected by rel_type. By default, all relationships point towards the right node. Store.save(subj, node=None): It save and creates a given entity/node—subj into the graph database. The second argument is of type Node, which if given will not create a new node and will change the already existing node. Store.save_indexed(index_name,key,value,subj): It saves the given entity into the graph and also creates an entry into the given index for future reference. Refer to http://py2neo.org/2.0/ext/ogm.html#py2neo.ext.ogm.Store for the complete list of methods exposed by Store class. Let's move on to the next section where we will use the OGM for mutating our social network data model. OGM supports Neo4j version 1.9, so all features of Neo4j 2.0 and above are not supported such as labels. Social network application with Flask-RESTful and OGM In this section, we will develop a full-fledged application for mutating our social network data and will also talk about the basics of Flask-RESTful and OGM. Creating object model Perform the following steps to create the object model and CRUD/search functions for our social network data: Our social network data contains two kind of entities—Person and Movies. So as a first step let's create a package model and within the model package let's define a module SocialDataModel.py with two classes—Person and Movie: class Person(object): def __init__(self, name=None,surname=None,age=None,country=None): self.name=name self.surname=surname self.age=age self.country=country class Movie(object): def __init__(self, movieName=None): self.movieName=movieName Next, let's define another package operations and two python modules ExecuteCRUDOperations.py and ExecuteSearchOperations.py. The ExecuteCRUDOperations module will contain the following three classes: DeleteNodesRelationships: It will contain one method each for deleting People nodes and Movie nodes and in the __init__ method, we will establish the connection to the graph database. class DeleteNodesRelationships(object): ''' Define the Delete Operation on Nodes ''' def __init__(self,host,port,username,password): #Authenticate and Connect to the Neo4j Graph Database py2neo.authenticate(host+':'+port, username, password) graph = Graph('http://'+host+':'+port+'/db/data/') store = Store(graph) #Store the reference of Graph and Store. self.graph=graph self.store=store def deletePersonNode(self,node): #Load the node from the Neo4j Legacy Index cls = self.store.load_indexed('personIndex', 'name', node.name, Person) #Invoke delete method of store class self.store.delete(cls[0]) def deleteMovieNode(self,node): #Load the node from the Neo4j Legacy Index cls = self.store.load_indexed('movieIndex', 'name',node.movieName, Movie) #Invoke delete method of store class self.store.delete(cls[0]) Deleting nodes will also delete the associated relationships, so there is no need to have functions for deleting relationships. Nodes without any relationship do not make much sense for many business use cases, especially in a social network, unless there is a specific need or an exceptional scenario. UpdateNodesRelationships: It will contain one method each for updating People nodes and Movie nodes and, in the __init__ method, we will establish the connection to the graph database. class UpdateNodesRelationships(object): ''' Define the Update Operation on Nodes ''' def __init__(self,host,port,username,password): #Write code for connecting to server def updatePersonNode(self,oldNode,newNode): #Get the old node from the Index cls = self.store.load_indexed('personIndex', 'name', oldNode.name, Person) #Copy the new values to the Old Node cls[0].name=newNode.name cls[0].surname=newNode.surname cls[0].age=newNode.age cls[0].country=newNode.country #Delete the Old Node form Index self.store.delete(cls[0]) #Persist the updated values again in the Index self.store.save_unique('personIndex', 'name', newNode.name, cls[0]) def updateMovieNode(self,oldNode,newNode): #Get the old node from the Index cls = self.store.load_indexed('movieIndex', 'name', oldNode.movieName, Movie) #Copy the new values to the Old Node cls[0].movieName=newNode.movieName #Delete the Old Node form Index self.store.delete(cls[0]) #Persist the updated values again in the Index self.store.save_ unique('personIndex', 'name', newNode.name, cls[0]) CreateNodesRelationships: This class will contain methods for creating People and Movies nodes and relationships and will then further persist them to the database. As with the other classes/ module, it will establish the connection to the graph database in the __init__ method: class CreateNodesRelationships(object): ''' Define the Create Operation on Nodes ''' def __init__(self,host,port,username,password): #Write code for connecting to server ''' Create a person and store it in the Person Dictionary. Node is not saved unless save() method is invoked. Helpful in bulk creation ''' def createPerson(self,name,surName=None,age=None,country=None): person = Person(name,surName,age,country) return person ''' Create a movie and store it in the Movie Dictionary. Node is not saved unless save() method is invoked. Helpful in bulk creation ''' def createMovie(self,movieName): movie = Movie(movieName) return movie ''' Create a relationships between 2 nodes and invoke a local method of Store class. Relationship is not saved unless Node is saved or save() method is invoked. ''' def createFriendRelationship(self,startPerson,endPerson): self.store.relate(startPerson, 'FRIEND', endPerson) ''' Create a TEACHES relationships between 2 nodes and invoke a local method of Store class. Relationship is not saved unless Node is saved or save() method is invoked. ''' def createTeachesRelationship(self,startPerson,endPerson): self.store.relate(startPerson, 'TEACHES', endPerson) ''' Create a HAS_RATED relationships between 2 nodes and invoke a local method of Store class. Relationship is not saved unless Node is saved or save() method is invoked. ''' def createHasRatedRelationship(self,startPerson,movie,ratings): self.store.relate(startPerson, 'HAS_RATED', movie,{'ratings':ratings}) ''' Based on type of Entity Save it into the Server/ database ''' def save(self,entity,node): if(entity=='person'): self.store.save_unique('personIndex', 'name', node.name, node) else: self.store.save_unique('movieIndex','name',node.movieName,node) Next we will define other Python module operations, ExecuteSearchOperations.py. This module will define two classes, each containing one method for searching Person and Movie node and of-course the __init__ method for establishing a connection with the server: class SearchPerson(object): ''' Class for Searching and retrieving the the People Node from server ''' def __init__(self,host,port,username,password): #Write code for connecting to server def searchPerson(self,personName): cls = self.store.load_indexed('personIndex', 'name', personName, Person) return cls; class SearchMovie(object): ''' Class for Searching and retrieving the the Movie Node from server ''' def __init__(self,host,port,username,password): #Write code for connecting to server def searchMovie(self,movieName): cls = self.store.load_indexed('movieIndex', 'name', movieName, Movie) return cls; We are done with our data model and the utility classes that will perform the CRUD and search operation over our social network data using py2neo OGM. Now let's move on to the next section and develop some REST services over our data model. Creating REST APIs over data models In this section, we will create and expose REST services for mutating and searching our social network data using the data model created in the previous section. In our social network data model, there will be operations on either the Person or Movie nodes, and there will be one more operation which will define the relationship between Person and Person or Person and Movie. So let's create another package service and define another module MutateSocialNetworkDataService.py. In this module, apart from regular imports from flask and flask_restful, we will also import classes from our custom packages created in the previous section and create objects of model classes for performing CRUD and search operations. Next we will define the different classes or services which will define the structure of our REST Services. The PersonService class will define the GET, POST, PUT, and DELETE operations for searching, creating, updating, and deleting the Person nodes. class PersonService(Resource): ''' Defines operations with respect to Entity - Person ''' #example - GET http://localhost:5000/person/Bradley def get(self, name): node = searchPerson.searchPerson(name) #Convert into JSON and return it back return jsonify(name=node[0].name,surName=node[0].surname,age=node[0].age,country=node[0].country) #POST http://localhost:5000/person #{"name": "Bradley","surname": "Green","age": "24","country": "US"} def post(self): jsonData = request.get_json(cache=False) attr={} for key in jsonData: attr[key]=jsonData[key] print(key,' = ',jsonData[key] ) person = createOperation.createPerson(attr['name'],attr['surname'],attr['age'],attr['country']) createOperation.save('person',person) return jsonify(result='success') #POST http://localhost:5000/person/Bradley #{"name": "Bradley1","surname": "Green","age": "24","country": "US"} def put(self,name): oldNode = searchPerson.searchPerson(name) jsonData = request.get_json(cache=False) attr={} for key in jsonData: attr[key] = jsonData[key] print(key,' = ',jsonData[key] ) newNode = Person(attr['name'],attr['surname'],attr['age'],attr['country']) updateOperation.updatePersonNode(oldNode[0],newNode) return jsonify(result='success') #DELETE http://localhost:5000/person/Bradley1 def delete(self,name): node = searchPerson.searchPerson(name) deleteOperation.deletePersonNode(node[0]) return jsonify(result='success') The MovieService class will define the GET, POST, and DELETE operations for searching, creating, and deleting the Movie nodes. This service will not support the modification of Movie nodes because, once the Movie node is defined, it does not change in our data model. Movie service is similar to our Person service and leverages our data model for performing various operations. The RelationshipService class only defines POST which will create the relationship between the person and other given entity and can either be another Person or Movie. Following is the structure of the POST method: ''' Assuming that the given nodes are already created this operation will associate Person Node either with another Person or Movie Node. Request for Defining relationship between 2 persons: - POST http://localhost:5000/relationship/person/Bradley {"entity_type":"person","person.name":"Matthew","relationship": "FRIEND"} Request for Defining relationship between Person and Movie POST http://localhost:5000/relationship/person/Bradley {"entity_type":"Movie","movie.movieName":"Avengers","relationship": "HAS_RATED" "relationship.ratings":"4"} ''' def post(self, entity,name): jsonData = request.get_json(cache=False) attr={} for key in jsonData: attr[key]=jsonData[key] print(key,' = ',jsonData[key] ) if(entity == 'person'): startNode = searchPerson.searchPerson(name) if(attr['entity_type']=='movie'): endNode = searchMovie.searchMovie(attr['movie.movieName']) createOperation.createHasRatedRelationship(startNode[0], endNode[0], attr['relationship.ratings']) createOperation.save('person', startNode[0]) elif (attr['entity_type']=='person' and attr['relationship']=='FRIEND'): endNode = searchPerson.searchPerson(attr['person.name']) createOperation.createFriendRelationship(startNode[0], endNode[0]) createOperation.save('person', startNode[0]) elif (attr['entity_type']=='person' and attr['relationship']=='TEACHES'): endNode = searchPerson.searchPerson(attr['person.name']) createOperation.createTeachesRelationship(startNode[0], endNode[0]) createOperation.save('person', startNode[0]) else: raise HTTPException("Value is not Valid") return jsonify(result='success') At the end, we will define our __main__ method, which will bind our services with the specific URLs and bring up our application: if __name__ == '__main__': api.add_resource(PersonService,'/person','/person/<string:name>') api.add_resource(MovieService,'/movie','/movie/<string:movieName>') api.add_resource(RelationshipService,'/relationship','/relationship/<string:entity>/<string:name>') webapp.run(debug=True) And we are done!!! Execute our MutateSocialNetworkDataService.py as a regular Python module and your REST-based services are up and running. Users of this app can use any REST-based clients such as SOAP-UI and can execute the various REST services for performing CRUD and search operations. Follow the comments provided in the code samples for the format of the request/response. In this section, we created and exposed REST-based services using Flask, Flask-RESTful, and OGM and performed CRUD and search operations over our social network data model. Using Neomodel in a Django app In this section, we will talk about the integration of Django and Neomodel. Django is a Python-based, powerful, robust, and scalable web-based application development framework. It is developed upon the Model-View-Controller (MVC) design pattern where developers can design and develop a scalable enterprise-grade application within no time. We will not go into the details of Django as a web-based framework but will assume that the readers have a basic understanding of Django and some hands-on experience in developing web-based and database-driven applications. Visit https://docs.djangoproject.com/en/1.7/ if you do not have any prior knowledge of Django. Django provides various signals or triggers that are activated and used to invoke or execute some user-defined functions on a particular event. The framework invokes various signals or triggers if there are any modifications requested to the underlying application data model such as pre_save(), post_save(), pre_delete, post_delete, and a few more. All the functions starting with pre_ are executed before the requested modifications are applied to the data model, and functions starting with post_ are triggered after the modifications are applied to the data model. And that's where we will hook our Neomodel framework, where we will capture these events and invoke our custom methods to make similar changes to our Neo4j database. We can reuse our social data model and the functions defined in ExploreSocialDataModel.CreateDataModel. We only need to register our event and things will be automatically handled by the Django framework. For example, you can register for the event in your Django model (models.py) by defining the following statement: signals.pre_save.connect(preSave, sender=Male) In the previous statement, preSave is the custom or user-defined method, declared in models.py. It will be invoked before any changes are committed to entity Male, which is controlled by the Django framework and is different from our Neomodel entity. Next, in preSave you need to define the invocations to the Neomodel entities and save them. Refer to the documentation at https://docs.djangoproject.com/en/1.7/topics/signals/ for more information on implementing signals in Django. Signals in Neomodel Neomodel also provides signals that are similar to Django signals and have the same behavior. Neomodel provides the following signals: pre_save, post_save, pre_delete, post_delete, and post_create. Neomodel exposes the following two different approaches for implementing signals: Define the pre..() and post..() methods in your model itself and Neomodel will automatically invoke it. For example, in our social data model, we can define def pre_save(self) in our Model.Male class to receive all events before entities are persisted in the database or server. Another approach is using Django-style signals, where we can define the connect() method in our Neomodel Model.py and it will produce the same results as in Django-based models: signals.pre_save.connect(preSave, sender=Male) Refer to http://neomodel.readthedocs.org/en/latest/hooks.html for more information on signals in Neomodel. In this section, we discussed about the integration of Django with Neomodel using Django signals. We also talked about the signals provided by Neomodel and their implementation approach. Summary Here we learned about creating web-based applications using Flask. We also used Flasks extensions such as Flask-RESTful for creating/exposing REST APIs for data manipulation. Finally, we created a full blown REST-based application over our social network data using Flask, Flask-RESTful, and py2neo OGM. We also learned about Neomodel and its various features and APIs provided to work with Neo4j. We also discussed about the integration of Neomodel with the Django framework. Resources for Article: Further resources on this subject: Firebase [article] Developing Location-based Services with Neo4j [article] Learning BeagleBone Python Programming [article]

0
0
6793

How-To Tutorials

Packt

21 Sep 2015

18 min read

Scraping the Data

Packt

21 Sep 2015

18 min read

In this article by Richard Lawson, author of the book Web Scraping with Python, we will first cover a browser extension called Firebug Lite to examine a web page, which you may already be familiar with if you have a web development background. Then, we will walk through three approaches to extract data from a web page using regular expressions, Beautiful Soup and lxml. Finally, the article will conclude with a comparison of these three scraping alternatives. (For more resources related to this topic, see here.) Analyzing a web page To understand how a web page is structured, we can try examining the source code. In most web browsers, the source code of a web page can be viewed by right-clicking on the page and selecting the View page source option: The data we are interested in is found in this part of the HTML: <table> <tr id="places_national_flag__row"><td class="w2p_fl"><label for="places_national_flag" id="places_national_flag__label">National Flag: </label></td><td class="w2p_fw"><img src="/places/static/images/flags/gb.png" /></td><td class="w2p_fc"></td></tr> … <tr id="places_neighbours__row"><td class="w2p_fl"><label for="places_neighbours" id="places_neighbours__label">Neighbours: </label></td><td class="w2p_fw"><div><a href="/iso/IE">IE </a></div></td><td class="w2p_fc"></td></tr></table> This lack of whitespace and formatting is not an issue for a web browser to interpret, but it is difficult for us. To help us interpret this table, we will use the Firebug Lite extension, which is available for all web browsers at https://getfirebug.com/firebuglite. Firefox users can install the full Firebug extension if preferred, but the features we will use here are included in the Lite version. Now, with Firebug Lite installed, we can right-click on the part of the web page we are interested in scraping and select Inspect with Firebug Lite from the context menu, as shown here: This will open a panel showing the surrounding HTML hierarchy of the selected element: In the preceding screenshot, the country attribute was clicked on and the Firebug panel makes it clear that the country area figure is included within a <td> element of class w2p_fw, which is the child of a <tr> element of ID places_area__row. We now have all the information needed to scrape the area data. Three approaches to scrape a web page Now that we understand the structure of this web page we will investigate three different approaches to scraping its data, firstly with regular expressions, then with the popular BeautifulSoup module, and finally with the powerful lxml module. Regular expressions If you are unfamiliar with regular expressions or need a reminder, there is a thorough overview available at https://docs.python.org/2/howto/regex.html. To scrape the area using regular expressions, we will first try matching the contents of the <td> element, as follows: >>> import re >>> url = 'http://example.webscraping.com/view/United Kingdom-239' >>> html = download(url) >>> re.findall('<td class="w2p_fw">(.*?)</td>', html) ['<img src="/places/static/images/flags/gb.png" />', '244,820 square kilometres', '62,348,447', 'GB', 'United Kingdom', 'London', '<a href="/continent/EU">EU</a>', '.uk', 'GBP', 'Pound', '44', '@# #@@|@## #@@|@@# #@@|@@## #@@|@#@ #@@|@@#@ #@@|GIR0AA', '^(([A-Z]\d{2}[A-Z]{2})|([A-Z]\d{3}[A-Z]{2})|([A-Z]{2}\d{2} [A-Z]{2})|([A-Z]{2}\d{3}[A-Z]{2})|([A-Z]\d[A-Z]\d[A-Z]{2}) |([A-Z]{2}\d[A-Z]\d[A-Z]{2})|(GIR0AA))$', 'en-GB,cy-GB,gd', '<div><a href="/iso/IE">IE </a></div>'] This result shows that the <td class="w2p_fw"> tag is used for multiple country attributes. To isolate the area, we can select the second element, as follows: >>> re.findall('<td class="w2p_fw">(.*?)</td>', html)[1] '244,820 square kilometres' This solution works but could easily fail if the web page is updated. Consider if the website is updated and the population data is no longer available in the second table row. If we just need to scrape the data now, future changes can be ignored. However, if we want to rescrape this data in future, we want our solution to be as robust against layout changes as possible. To make this regular expression more robust, we can include the parent <tr> element, which has an ID, so it ought to be unique: >>> re.findall('<tr id="places_area__row"><td class="w2p_fl"><label for="places_area" id="places_area__label">Area: </label></td><td class="w2p_fw">(.*?)</td>', html) ['244,820 square kilometres'] This iteration is better; however, there are many other ways the web page could be updated in a way that still breaks the regular expression. For example, double quotation marks might be changed to single, extra space could be added between the <td> tags, or the area_label could be changed. Here is an improved version to try and support these various possiblilities: >>> re.findall('<tr id="places_area__row">.*?<tds*class=["']w2p_fw["']>(.*?) </td>', html)[0] '244,820 square kilometres' This regular expression is more future-proof but is difficult to construct, becoming unreadable. Also, there are still other minor layout changes that would break it, such as if a title attribute was added to the <td> tag. From this example, it is clear that regular expressions provide a simple way to scrape data but are too brittle and will easily break when a web page is updated. Fortunately, there are better solutions. Beautiful Soup Beautiful Soup is a popular library that parses a web page and provides a convenient interface to navigate content. If you do not already have it installed, the latest version can be installed using this command: pip install beautifulsoup4 The first step with Beautiful Soup is to parse the downloaded HTML into a soup document. Most web pages do not contain perfectly valid HTML and Beautiful Soup needs to decide what is intended. For example, consider this simple web page of a list with missing attribute quotes and closing tags: <ul class=country> <li>Area <li>Population </ul> If the Population item is interpreted as a child of the Area item instead of the list, we could get unexpected results when scraping. Let us see how Beautiful Soup handles this: >>> from bs4 import BeautifulSoup >>> broken_html = '<ul class=country><li>Area<li>Population</ul>' >>> # parse the HTML >>> soup = BeautifulSoup(broken_html, 'html.parser') >>> fixed_html = soup.prettify() >>> print fixed_html <html> <body> <ul class="country"> <li>Area</li> <li>Population</li> </ul> </body> </html> Here, BeautifulSoup was able to correctly interpret the missing attribute quotes and closing tags, as well as add the <html> and <body> tags to form a complete HTML document. Now, we can navigate to the elements we want using the find() and find_all() methods: >>> ul = soup.find('ul', attrs={'class':'country'}) >>> ul.find('li') # returns just the first match <li>Area</li> >>> ul.find_all('li') # returns all matches [<li>Area</li>, <li>Population</li>] Beautiful Soup overview Here are the common methods and parameters you will use when scraping web pages with Beautiful Soup: BeautifulSoup(markup, builder): This method creates the soup object. The markup parameter can be a string or file object, and builder is the library that parses the markup parameter. find_all(name, attrs, text, **kwargs): This method returns a list of elements matching the given tag name, dictionary of attributes, and text. The contents of kwargs are used to match attributes. find(name, attrs, text, **kwargs): This method is the same as find_all(), except that it returns only the first match. If no element matches, it returns None. prettify(): This method returns the parsed HTML in an easy-to-read format with indentation and line breaks. For a full list of available methods and parameters, the official documentation is available at http://www.crummy.com/software/BeautifulSoup/bs4/doc/. Now, using these techniques, here is a full example to extract the area from our example country: >>> from bs4 import BeautifulSoup >>> url = 'http://example.webscraping.com/places/view/ United-Kingdom-239' >>> html = download(url) >>> soup = BeautifulSoup(html) >>> # locate the area row >>> tr = soup.find(attrs={'id':'places_area__row'}) >>> td = tr.find(attrs={'class':'w2p_fw'}) # locate the area tag >>> area = td.text # extract the text from this tag >>> print area 244,820 square kilometres This code is more verbose than regular expressions but easier to construct and understand. Also, we no longer need to worry about problems in minor layout changes, such as extra whitespace or tag attributes. Lxml Lxml is a Python wrapper on top of the libxml2 XML parsing library written in C, which makes it faster than Beautiful Soup but also harder to install on some computers. The latest installation instructions are available at http://lxml.de/installation.html. As with Beautiful Soup, the first step is parsing the potentially invalid HTML into a consistent format. Here is an example of parsing the same broken HTML: >>> import lxml.html >>> broken_html = '<ul class=country><li>Area<li>Population</ul>' >>> tree = lxml.html.fromstring(broken_html) # parse the HTML >>> fixed_html = lxml.html.tostring(tree, pretty_print=True) >>> print fixed_html <ul class="country"> <li>Area</li> <li>Population</li> </ul> As with BeautifulSoup, lxml was able to correctly parse the missing attribute quotes and closing tags, although it did not add the <html> and <body> tags. After parsing the input, lxml has a number of different options to select elements, such as XPath selectors and a find() method similar to Beautiful Soup. Instead, we will use CSS selectors here and in future examples, because they are more compact. Also, some readers will already be familiar with them from their experience with jQuery selectors. Here is an example using the lxml CSS selectors to extract the area data: >>> tree = lxml.html.fromstring(html) >>> td = tree.cssselect('tr#places_area__row > td.w2p_fw')[0] >>> area = td.text_content() >>> print area 244,820 square kilometres The key line with the CSS selector is highlighted. This line finds a table row element with the places_area__row ID, and then selects the child table data tag with the w2p_fw class. CSS selectors CSS selectors are patterns used for selecting elements. Here are some examples of common selectors you will need: Select any tag: * Select by tag <a>: a Select by class of "link": .link Select by tag <a> with class "link": a.link Select by tag <a> with ID "home": a#home Select by child <span> of tag <a>: a > span Select by descendant <span> of tag <a>: a span Select by tag <a> with attribute title of "Home": a[title=Home] The CSS3 specification was produced by the W3C and is available for viewing at http://www.w3.org/TR/2011/REC-css3-selectors-20110929/. Lxml implements most of CSS3, and details on unsupported features are available at https://pythonhosted.org/cssselect/#supported-selectors. Note that, internally, lxml converts the CSS selectors into an equivalent XPath. Comparing performance To help evaluate the trade-offs of the three scraping approaches described in this article, it would help to compare their relative efficiency. Typically, a scraper would extract multiple fields from a web page. So, for a more realistic comparison, we will implement extended versions of each scraper that extract all the available data from a country's web page. To get started, we need to return to Firebug to check the format of the other country features, as shown here: Firebug shows that each table row has an ID starting with places_ and ending with __row. Then, the country data is contained within these rows in the same format as the earlier area example. Here are implementations that use this information to extract all of the available country data: FIELDS = ('area', 'population', 'iso', 'country', 'capital', 'continent', 'tld', 'currency_code', 'currency_name', 'phone', 'postal_code_format', 'postal_code_regex', 'languages', 'neighbours') import re def re_scraper(html): results = {} for field in FIELDS: results[field] = re.search('<tr id="places_%s__row">.*?<td class="w2p_fw">(.*?)</td>' % field, html).groups()[0] return results from bs4 import BeautifulSoup def bs_scraper(html): soup = BeautifulSoup(html, 'html.parser') results = {} for field in FIELDS: results[field] = soup.find('table').find('tr', id='places_%s__row' % field).find('td', class_='w2p_fw').text return results import lxml.html def lxml_scraper(html): tree = lxml.html.fromstring(html) results = {} for field in FIELDS: results[field] = tree.cssselect('table > tr#places_%s__row > td.w2p_fw' % field)[0].text_content() return results Scraping results Now that we have complete implementations for each scraper, we will test their relative performance with this snippet: import time NUM_ITERATIONS = 1000 # number of times to test each scraper html = download('http://example.webscraping.com/places/view/ United-Kingdom-239') for name, scraper in [('Regular expressions', re_scraper), ('BeautifulSoup', bs_scraper), ('Lxml', lxml_scraper)]: # record start time of scrape start = time.time() for i in range(NUM_ITERATIONS): if scraper == re_scraper: re.purge() result = scraper(html) # check scraped result is as expected assert(result['area'] == '244,820 square kilometres') # record end time of scrape and output the total end = time.time() print '%s: %.2f seconds' % (name, end – start) This example will run each scraper 1000 times, check whether the scraped results are as expected, and then print the total time taken. Note the highlighted line calling re.purge(); by default, the regular expression module will cache searches and this cache needs to be cleared to make a fair comparison with the other scraping approaches. Here are the results from this script on my computer: $ python performance.py Regular expressions: 5.50 seconds BeautifulSoup: 42.84 seconds Lxml: 7.06 seconds The results on your computer will quite likely be different because of the different hardware used. However, the relative difference between each approach should be equivalent. The results show that Beautiful Soup is over six times slower than the other two approaches when used to scrape our example web page. This result could be anticipated because lxml and the regular expression module were written in C, while BeautifulSoup is pure Python. An interesting fact is that lxml performed comparatively well with regular expressions, since lxml has the additional overhead of having to parse the input into its internal format before searching for elements. When scraping many features from a web page, this initial parsing overhead is reduced and lxml becomes even more competitive. It really is an amazing module! Overview The following table summarizes the advantages and disadvantages of each approach to scraping: Scraping approach Performance Ease of use Ease to install Regular expressions Fast Hard Easy (built-in module) Beautiful Soup Slow Easy Easy (pure Python) Lxml Fast Easy Moderately difficult If the bottleneck to your scraper is downloading web pages rather than extracting data, it would not be a problem to use a slower approach, such as Beautiful Soup. Or, if you just need to scrape a small amount of data and want to avoid additional dependencies, regular expressions might be an appropriate choice. However, in general, lxml is the best choice for scraping, because it is fast and robust, while regular expressions and Beautiful Soup are only useful in certain niches. Adding a scrape callback to the link crawler Now that we know how to scrape the country data, we can integrate this into the link crawler. To allow reusing the same crawling code to scrape multiple websites, we will add a callback parameter to handle the scraping. A callback is a function that will be called after certain events (in this case, after a web page has been downloaded). This scrape callback will take a url and html as parameters and optionally return a list of further URLs to crawl. Here is the implementation, which is simple in Python: def link_crawler(..., scrape_callback=None): … links = [] if scrape_callback: links.extend(scrape_callback(url, html) or []) … The new code for the scraping callback function are highlighted in the preceding snippet. Now, this crawler can be used to scrape multiple websites by customizing the function passed to scrape_callback. Here is a modified version of the lxml example scraper that can be used for the callback function: def scrape_callback(url, html): if re.search('/view/', url): tree = lxml.html.fromstring(html) row = [tree.cssselect('table > tr#places_%s__row > td.w2p_fw' % field)[0].text_content() for field in FIELDS] print url, row This callback function would scrape the country data and print it out. Usually, when scraping a website, we want to reuse the data, so we will extend this example to save results to a CSV spreadsheet, as follows: import csv class ScrapeCallback: def __init__(self): self.writer = csv.writer(open('countries.csv', 'w')) self.fields = ('area', 'population', 'iso', 'country', 'capital', 'continent', 'tld', 'currency_code', 'currency_name', 'phone', 'postal_code_format', 'postal_code_regex', 'languages', 'neighbours') self.writer.writerow(self.fields) def __call__(self, url, html): if re.search('/view/', url): tree = lxml.html.fromstring(html) row = [] for field in self.fields: row.append(tree.cssselect('table > tr#places_{}__row > td.w2p_fw'.format(field)) [0].text_content()) self.writer.writerow(row) To build this callback, a class was used instead of a function so that the state of the csv writer could be maintained. This csv writer is instantiated in the constructor, and then written to multiple times in the __call__ method. Note that __call__ is a special method that is invoked when an object is "called" as a function, which is how the cache_callback is used in the link crawler. This means that scrape_callback(url, html) is equivalent to calling scrape_callback.__call__(url, html). For further details on Python's special class methods, refer to https://docs.python.org/2/reference/datamodel.html#special-method-names. This code shows how to pass this callback to the link crawler: link_crawler('http://example.webscraping.com/', '/(index|view)', max_depth=-1, scrape_callback=ScrapeCallback()) Now, when the crawler is run with this callback, it will save results to a CSV file that can be viewed in an application such as Excel or LibreOffice: Success! We have completed our first working scraper. Summary In this article, we walked through a variety of ways to scrape data from a web page. Regular expressions can be useful for a one-off scrape or to avoid the overhead of parsing the entire web page, and BeautifulSoup provides a high-level interface while avoiding any difficult dependencies. However, in general, lxml will be the best choice because of its speed and extensive functionality, and we will use it in future examples. Resources for Article: Further resources on this subject: Scientific Computing APIs for Python [article] Bizarre Python [article] Optimization in Python [article]

0
0
6792

article-image-customizing-layout-themes-php-nuke

Packt

31 Mar 2010

23 min read

Customizing Layout with Themes in PHP-Nuke

Packt

31 Mar 2010

23 min read

Creating a PHP-Nuke theme gives your site its own special look, distinguishing it from other PHP-Nuke-created sites and offers an effective outlet for your creative talents. Creating a theme requires some knowledge of HTML, confidence in working with CSS and PHP, but most important is some imagination and creativity! Unlike the tasks we have tackled in previous articles, where we have been working exclusively through a web browser to control and configure PHP-Nuke, working with themes is the start of a new era in your PHP-Nuke skills; editing the code files of PHP-Nuke itself. Fortunately, the design of PHP-Nuke means that our theme work won't be tampering with the inner workings of PHP-Nuke. However, becoming confident in handling the mixture of HTML and PHP code that is a PHP-Nuke theme will prepare you for the more advanced work ahead, when we really get to grips with PHP-Nuke at the code level. In this article, we will look at: Theme management Templates in themes Changing the page header Working with the stylesheet Changing blocks Changing the format of stories What Does a Theme Control? Despite the fact that we say 'themes control the look and feel of your site', a theme does not determine every aspect of the page output. PHP-Nuke is an incredibly versatile application, but it cannot produce every website imaginable. Appearance First of all, the appearance of the page can be controlled through the use of colors, fonts, font sizes, weights, and so on. This can either be done through the use of CSS styles or HTML. You can also add JavaScript for fancier effects, or even Flash animations, Java applets, or sounds—anything that you can add to a standard HTML page. Graphical aspects of the page such as the site banner, background images, and so on, are under the care of the theme. There are also some modules that allow their standard graphical icons to be overridden with images from a theme. Page Layout Roughly speaking, a PHP-Nuke page consists of three parts; the top bit, the bit in the middle, and the bit at the bottom! The top bit—the header—usually contains a site logo and such things as a horizontal navigation bar for going directly to important parts of your site. The bottom bit—the footer—contains the copyright message. In between the header and the footer, the output is usually divided into three columns. The left-hand column typically contains blocks, displayed one of top each other, the middle column contains the module output, and the right-hand column contains more blocks. The layout of these columns (their width for example) is controlled by the theme. You may have noticed that the right-hand column is generally only displayed on the homepage of a PHP-Nuke site; this too, is something that is controlled by the theme. The appearance of the blocks is controlled by the theme; PHP-Nuke provides the title of the block and its content, and the theme will generally 'frame' these to produce the familiar block look. The theme also determines how the description of stories appears on the homepage. In addition, the theme determines how the full text of the story, its extended text, is displayed. We've talked about how the theme controls the 'look' of things. The theme also allows you to add other site-related data to your page; for example the name of the site can appear, and the site slogan, and you can even add such things as the user's name with a friendly welcome message. Theme Management Basically, a theme is a folder that sits inside the themes folder in your PHP-Nuke installation. Different themes correspond to different folders in the themes folder, and adding or removing a theme is as straightforward as adding or removing the relevant folder from the themes folder. By default, you will find around 14 themes in a standard PHP-Nuke installation. DeepBlue is the default theme. Themes can be chosen in one of two ways: By the administrator: You can simply select the required theme from the General Site Info panel of the Site Preferences administration menu and save the changes. The theme selected by the administrator is the default theme for the site and will be seen by all users of the site, registered or unregistered. By the user: Users can override the default theme set by the administrator from the Themes option of the Your Account module. This sets a new, personal, theme that will be displayed to that user. Note that this isn't a theme especially customized for that user; it is just one chosen from the list of standard themes installed on your site. Unregistered visitors do not have an option to choose a theme; they have to become registered users. Theme File Structure Let's start with the default theme, DeepBlue. If you open up the DeepBlue folder within the themes folder in the root of your PHP-Nuke installation, you will find three folders and two files. The three folders are: forums: This folder contains the theme for the Forums module. This is not strictly a requirement of a PHP-Nuke theme, and not every PHP-Nuke theme has a forums theme. The Forums module (otherwise known as phpBB) has its own theme 'engine'. The purpose of including a theme for the forums is that you have consistency between the rest of your PHP-Nuke display and the phpBB display. images: This folder contains the image files used by your theme. These include the site logo, background images, and graphics for blocks among others. As mentioned earlier, within this folder can be other folders containing images to override the standard icons. style: This folder contains the CSS files for your theme. Usually, there is one CSS file in the style folder, style.css. Each theme will make use of its style.css file, and this is the file into which we will add our style definitions when the time comes. Of the two files, index.html is simply there to prevent people browsing to your themes folder and seeing what it contains; visiting this page in a browser simply produces a blank page. It is a very simple security measure. The themes.php file is a PHP code file, and is where all the action happens. This file must always exist within a theme folder. We will concentrate on this file later when we customize the theme. In other themes you will find more files; we will look at these later. Installing a New Theme Installing and uninstalling themes comes down to adding or removing folders from the themes folder, and whenever a list of available themes is presented, either in the Site Preferences menu or the Your Accounts module, PHP-Nuke refreshes this list by getting the names of the folders in the themes folder. You will find a huge range of themes on the Web. For example, there is a gallery of themes at: http://nukecops.com/modules.php?set_albumName=packs&op=modload&name=Gallery& file=index&include=view_album.php Many of these are themes written for older versions of PHP-Nuke, but most are still compatible with the newer releases. There is also a live demonstration of some themes at: http://www.portedmods.com/styles/ On this page you can select the new theme and see it applied immediately, before you download it. Removing an Existing Theme To remove a theme from your PHP-Nuke site you simply remove the corresponding folder from the themes folder, and it will no longer be available to PHP-Nuke. However, you should be careful when removing themes—what if somebody is actually using that theme? If a user has that theme selected as their personal theme, and you remove that theme, then that user's personal theme will revert to the default theme selected in Site Preferences. If you remove the site's default theme, then you will break your site! Deleting the site's default theme will produce either a blank screen or messages like the following when you attempt to view your site. Warning: head(themes/NonExistentTheme/theme.php)[function.head]: failed to create stream:No such file or directory in c:nukehtmlheader.php on line 31 The only people who can continue to use your site in this situation are those who have selected a personal theme for themselves—and only if that theme is still installed. To correct such a faux pas, make a copy of one of the other themes in your themes folder (unless you happen to have a copy of the theme you just deleted elsewhere), and rename it to the name of the theme you just deleted. In conclusion, removing themes should only be a problem if you somehow manage to remove your site's default theme. For users who have selected the theme you just removed, their theme will revert to the default theme and life goes on for them. A final caveat about the names of theme folders; do not use spaces in the names of the folders in the themes folder—this can lead to strange behavior when the list of themes is displayed in the drop-down menus for users to select from. From an Existing Theme to a New Theme We'll create a new theme for the Dinosaur Portal by making changes to an existing theme. This will not only make you feel like the theme master, but it will also serve to illustrate the nature of the theme-customization problem. We'll be making changes all over the place—adding and replacing things in HTML and PHP files—but it will be worth it. Another thing to bear in mind is that we're creating a completely different looking site without making any changes to the inner parts of PHP-Nuke. At this point, all we are changing is the theme definition. The theme for the Dinosaur Portal will have a warm, tropical feel to it to evoke the atmosphere of a steaming, tropical, prehistoric jungle, and will use lots of orange color on the page. First of all, we need a theme on which to conduct our experiments. We'll work on the 3D-Fantasy theme. Starting Off The first thing we will do is to create a new theme folder, which will be a copy of the 3D-Fantasy theme. Open up the themes folder in your file explorer, and create a copy of the 3D-Fantasy folder. Rename this copy as TheDinosaurPortal. Now log into your site as testuser, and from the Your Account module, select TheDinosaurPortal as the theme. Your site will immediately switch to this theme, but it will look exactly like 3D-Fantasy, because, at the moment, it is! You will also need some images from the code download for this article; you will find them in the SiteImages folder of this article's code. Replacing Traces of the Old Theme The theme that we are about to work on has many occurrences of 3D-Fantasy in a number of files, such as references to images. We will have to remove these first of all, or else our new theme will be looking in the wrong folder for images and other resources. Open each of the files below in your text editor, and replace every occurrence of 3D-Fantasy with TheDinosaurPortal in a text editor, we'll use Wordpad. "You can use the replace functionality of your editor to do this. For example, in Wordpad, select Edit | Replace; enter the text to be replaced, and then click on Replace All to replace all the occurrences in the open file. After making all the changes, save each file: blocks.html footer.html header.html story_home.html story_page.html theme.php tables.php Templates and PHP Files We've just encountered two types of file in the theme folder—PHP code files (theme.php and tables.php) and HTML files (blocks.html, footer.html, and so on). Before we go any further, we need to have a quick discussion of what roles these types of file play in the theme construction. PHP Files The PHP files do the main work of the theme. These files contain the definitions of some functions that handle the display of the page header and how an individual block or article is formatted, among other tasks. These functions are called from other parts of PHP-Nuke when required. We'll talk about them when they are required later in the article. Part of our customization work will be to make some changes to these functions and have them act in a different way when called. Historically, the code for a PHP-Nuke theme consisted of a single PHP file, theme.php. One major drawback of this was the difficulty you would have in editing this file in the 'design' view of an HTML editor. Instead of seeing the HTML that you wished to edit, you probably wouldn't see anything in the 'design' view of most HTML editors, since the HTML was inextricably intertwined with the PHP code. This made creating a new theme, or even editing an existing theme, not something for the faint-hearted—you had to be confident with your PHP coding to make sure you were changing the right places, and in the right way. The theme.php file consists of a number of functions that are called from other parts of PHP-Nuke when required. These functions are how the theme does its work. One of the neat appearances in recent versions of PHP-Nuke is the use of a 'mini-templating' engine for themes. Not all themes make use of this method (DeepBlue is one theme that doesn't), and that is one of the reasons we are working with 3D-Fantasy as our base theme, since it does follow the 'templating' model. Templates The HTML files that we modified above are the theme templates. They consist of HTML, without any PHP code. Each template is responsible for a particular part of the page, and is called into action by the functions of the theme when required. One advantage of using these templates is that they can be easily edited in visual HTML editors, such as Macromedia's Dreamweaver, without any PHP code to interfere with the page design. Another advantage of using these templates is to separate logic from presentation. The idea of a template is that it should determine how something is displayed (its presentation). The template makes use of some data supplied to it, but acquiring and choosing this data (the logic) is not done in the template. The template is processed or evaluated by the 'template engine', and output is generated. The template engine in this case is the theme.php file. To see how the template and PHP-Nuke 'communicate', let's look at an extract from the header.html file in the 3D-Fantasy folder: <a href="index.php"> <img src="themes/3D-Fantasy/images/logo.gif" border="0" alt="Welcome to $sitename" align="left"></a> The $sitename text (shown highlighted) is an example of what we'll call a placeholder. There is a correspondence between these placeholders and PHP variables that have the same name as the placeholder text. Themes that make use of this templating process more or less replace any text beginning with $ in the template by the value of the corresponding PHP variable. This means that you can make use of variables from PHP-Nuke itself in your themes; these could be the name of your site ($sitename), your site slogan, or even information about the user. In fact, you can add your own PHP code to create a new variable, which you can then display from within one of the templates. To complete the discussion, we will look at how the templates are processed in PHP-Nuke. The code below is a snippet from one of the themeheader() function in the theme.php file. This particular snippet is taken from the 3D-Fantasy theme. function themeheader(){ global $user, $banners, $sitename, $slogan, $cookie, $prefix, $anonymous, $db;... code continues ....$tmpl_file = "themes/3D-Fantasy/header.html";$thefile = implode("", file($tmpl_file));$thefile = addslashes($thefile);$thefile = "$r_file="".$thefile."";";eval($thefile);print $r_file;... code continues .... The processing starts with the line where the $tmpl_file variable is defined. This variable is set to the file name of the template to be processed, in this case header.html. The next line grabs the content of the file as a string. Let's suppose the header.html file contained the text You're welcomed to $sitename, thanks for coming!. Then, continuing in the code above, the $thefile variable would eventually hold this: $r_file = " You're welcomed to $sitename, thanks for coming!"; This looks very much like a PHP statement, and that is exactly what PHP-Nuke is attempting to create. The eval() function executes the statement; it defines the variable $r_file as above. This is equivalent to putting this line straight into the code: $r_file = " You're welcomed to $sitename, thanks for coming!"; If this line were in the PHP code, the value of the $sitename variable will be inserted into the string, and this is exactly how the placeholders in the templates are replaced with the values of the corresponding PHP variables. This means that the placeholders in templates can only use variables accessible at the point in the code where the template is processed with the eval() function. This means any parameters passed to the function at the time—global variables that have been announced with the global statement or any variables local to the function that have been defined before the line with the eval() function. This does mean that you will have to study the function processing the template to see what variables are available. In the examples in this article we'll look at the most relevant variables. The templates do not allow for any form of 'computation' within them; you cannot use loops or call PHP functions. You do your computations 'outside' the template in the theme.php file, and the results are 'pulled' into the template and displayed from there. Now that we're familiar with what we're going to be working with, let's get started. Changing the Page Header The first port of call will be creating a new version of the page header. We will make these customizations: Changing the site logo graphic Changing the layout of the page header Adding a welcome message to the user, and displaying the user's avatar Adding a drop-down list of topics to the header Creating a navigation bar Time For Action—Changing the Site Logo Graphic Grab the <>ilogo.gif file from the SiteImages folder in the code download. Copy it to the themes/TheDinosaurPortal/images folder, overwriting the existing logo.gif file. Refresh the page in your browser. The logo will have changed! What Just Happened? The logo.gif file in the images folder is the site logo. We replaced it with a new banner, and immediately the change came into effect. Time For Action—Changing the Site Header Layout In the theme folder is a file called header.html. Open up this file in a text editor, we'll use Wordpad. Replace all the code in this file with the following: <table border="0" cellspacing="0" cellpadding="6" width="100%" bgcolor="#FFCC33"> <tr valign="middle"> <td width="60%" align="right" rowspan="2"> <a href="index.php"><img src="themes/$GLOBALS[ThemeSel]/images/logo.gif" border="1" alt="Welcome to $sitename"> </a></td> <td width="40%" colspan="2"> <p align="center"><b>WELCOME TO $sitename!</b></td> </tr> <tr> <td width="20%">GAP</td> <td width="20%">GAP</td> </tr></table>$public_msg<br><table cellpadding="0" cellspacing="0" width="99%" border="0" align="center" bgcolor="#ffffff"><tr><td bgcolor="#ffffff" valign="top"> Save the header.html file. Refresh your browser. The site header now looks like this: What Just Happened? The header.html file is the template responsible for formatting the site header. Changing this file will change the format of your site header. We simply created a table that displays the site logo in the left-hand column, a welcome message in the right-hand column, and under that, two GAPs that we will add more to in a moment. We set the background color of the table to an orange color (#FFCC33). We used the $sitename placeholder to display the name of the site from the template. Note that everything after the line:  in our new header.html file is from the original file. (The characters here denote an HTML comment that is not displayed in the browser). This is because the end of the header.html file starts a new table that will continue in other templates. If we had removed these lines, the page output would have been broken. There was another interesting thing we used in the template, the $GLOBALS[ThemeSel] placeholder: <a href="index.php"><img src="themes/$GLOBALS[ThemeSel]/images/logo.gif" ThemeSel is a global variable that holds the name of the current theme—it's either the default site theme or the user's chosen theme. Although it's a global variable, using just $ThemeSel in the template would give a blank, this is because it has not been declared as global by the function in PHP-Nuke that consumes the header.html template. However, all the global variables can be accessed through the $GLOBALS array, and using $GLOBALS[ThemeSel] accesses this particular global variable. Note that this syntax is different from the way you may usually access elements of the $GLOBALS array in PHP. You might use $GLOBALS['ThemeSel'] or $GLOBALS["ThemeSel"]. Neither of these work in the template so we have to use the form without the ' or ". Time For Action—Fixing and Adding the Topics List Next we'll add the list of topics as a drop-down box to the page header. The visitor will be able to select one of the topics from the box, and then the list of stories from that topic will be displayed to them through the News module. Also, the current topic will be selected in the drop-down box to avoid confusion. This task involves fixing some bugs in the current version of the 3D-Fantasy theme. First of all, open the theme.php file and find the following line in the themeheader() function definition: $topics_list = "<select name="topic" onChange='submit()'>n"; Replace this line with these two lines: global $new_topic;$topics_list = "<select name="new_topic" onChange='submit()'>n"; If you move a few lines down in the themeheader() function, you will find this line: if ($topicid==$topic) { $sel = "selected "; } Replace $topic with $new_topic in this line to get: if ($topicid==$new_topic) { $sel = "selected "; } Save the theme.php file. Open the header.html file in your text editor, and where the second GAP is, make the modifications as shown below: <td width="20%">GAP</td> <td width="20%"><form action="modules.php?name=News&new_topic" method="post"> Select a Topic:<br>$topics_list</select></form></td></tr></table> Save the header.html file. Refresh your browser. You will see the new drop-down box in your page header: What just Happened? The themeheader() function is the function in theme.php responsible for processing the header.html template, and outputting the page header. The $topics_list variable has already been created for us in the themeheader() function, and can be used from the header.html template. It is a string of HTML that defines an HTML select drop-down list consisting of the topic titles. However, the first few steps require us to make a change to the $topics_list variable, correcting the name of the select element and also using the correct variable to ensure the current topic (if any) is selected in the drop-down box. The select element needs to have the name of new_topic, so that the News module is able to identify which topic we're after. This is all done with the changes to the theme.php file. First, we add the global statement to access the $new_topic variable, before correcting the name of the select element: global $new_topic;$topics_list = "<select name="new_topic" onChange='submit()'>n"; The next change we made is to make sure we are looking for the $new_topic variable, not the $topic variable, which isn't even defined: if ($topicid==$new_topic) { $sel = "selected "; } Now the $topics_list variable is corrected, all we have to do is add a placeholder for this variable to the header.html template, and some more HTML around it. We added the placeholder for $topics_list to display the drop-down list, and a message to go with it encouraging the reader to select a topic into one of the GAP table cells we created in the new-look header. The list of topics will be contained in a form tag, and when the user selects a topic, the form will be posted back to the server to the News module, and the stories in the selected topic will be displayed. (The extra HTML that handles submitting the form is contained with the $topics_list variable.) <form action="modules.php?name=News" method="post">Select a Topic:<br>$topics_list All that remains now is to close the select tag—the tag was opened in the $topics_list variable but not closed—and then close the form tag: </select></form> When the page is displayed, this is the HTML that PHP-Nuke produces for the topics drop-down list: <form action="modules.php?name=News&new_topic" method="post">Select a Topic:<br><select name="topic" onChange='submit()'><option value="">All Topics</option><option value="1">The Dinosaur Portal</option><option value="2">Dinosuar Hunting</option></select></form>

0
0
6792

article-image-working-with-sparks-graph-processing-library-graphframes

Pravin Dhandre

11 Jan 2018

12 min read

Working with Spark’s graph processing library, GraphFrames

Pravin Dhandre

11 Jan 2018

12 min read

0
0
6789

Packt

14 Sep 2015

6 min read

Getting Started with Meteor

Packt

14 Sep 2015

6 min read

In this article, based on Marcelo Reyna's book Meteor Design Patterns, we will see that when you want to develop an application of any kind, you want to develop it fast. Why? Because the faster you develop, the better your return on investment will be (your investment is time, and the real cost is the money you could have produced with that time). There are two key ingredients ofrapid web development: compilers and patterns. Compilers will help you so that youdon’t have to type much, while patterns will increase the paceat which you solve common programming issues. Here, we will quick-start compilers and explain how they relate withMeteor, a vast but simple topic. The compiler we will be looking at isCoffeeScript. (For more resources related to this topic, see here.) CoffeeScriptfor Meteor CoffeeScript effectively replaces JavaScript. It is much faster to develop in CoffeeScript, because it simplifies the way you write functions, objects, arrays, logical statements, binding, and much more.All CoffeeScript files are saved with a .coffee extension. We will cover functions, objects, logical statements, and binding, since thisis what we will use the most. Objects and arrays CoffeeScriptgets rid of curly braces ({}), semicolons (;), and commas (,). This alone saves your fingers from repeating unnecessary strokes on the keyboard. CoffeeScript instead emphasizes on the proper use of tabbing. Tabbing will not only make your code more readable (you are probably doing it already), but also be a key factor inmaking it work. Let’s look at some examples: #COFFEESCRIPT toolbox = hammer:true flashlight:false Here, we are creating an object named toolbox that contains two keys: hammer and flashlight. The equivalent in JavaScript would be this: //JAVASCRIPT - OUTPUT var toolbox = { hammer:true, flashlight:false }; Much easier! As you can see, we have to tab to express that both the hammer and the flashlight properties are a part of toolbox. The word var is not allowed in CoffeeScript because CoffeeScript automatically applies it for you. Let’stakea look at how we would createan array: #COFFEESCRIPT drill_bits = [ “1/16 in” “5/64 in” “3/32 in” “7/64 in” ] //JAVASCRIPT – OUTPUT vardrill_bits; drill_bits = [“1/16 in”,”5/64 in”,”3/32 in”,”7/64 in”]; Here, we can see we don’t need any commas, but we do need brackets to determine that this is an array. Logical statements and operators CoffeeScript also removes a lot ofparentheses (()) in logical statements and functions. This makes the logic of the code much easier to understand at the first glance. Let’s look at an example: #COFFEESCRIPT rating = “excellent” if five_star_rating //JAVASCRIPT – OUTPUT var rating; if(five_star_rating){ rating = “excellent”; } In this example, we can clearly see thatCoffeeScript is easier to read and write.Iteffectively replaces all impliedparentheses in any logical statement. Operators such as &&, ||, and !== are replaced by words. Here is a list of the operators that you will be using the most: CoffeeScript JavaScript is === isnt !== not ! and && or || true, yes, on true false, no, off false @, this this Let's look at a slightly more complex logical statement and see how it compiles: #COFFEESCRIPT # Suppose that “this” is an object that represents a person and their physical properties if@eye_coloris “green” retina_scan = “passed” else retina_scan = “failed” //JAVASCRIPT - OUTPUT if(this.eye_color === “green”){ retina_scan = “passed”; } else { retina_scan = “failed”; } When using @eye_color to substitute for this.eye_color, notice that we do not need . Functions JavaScript has a couple of ways of creating functions. They look like this: //JAVASCRIPT //Save an anonymous function onto a variable varhello_world = function(){ console.log(“Hello World!”); } //Declare a function functionhello_world(){ console.log(“Hello World!”); } CoffeeScript uses ->instead of the function()keyword.The following example outputs a hello_world function: #COFFEESCRIPT #Create a function hello_world = -> console.log “Hello World!” //JAVASCRIPT - OUTPUT varhello_world; hello_world = function(){ returnconsole.log(“Hello World!”); } Once again, we use a tab to express the content of the function, so there is no need ofcurly braces ({}). This means that you have to make sure you have all of the logic of the function tabbed under its namespace. But what about our parameters? We can use (p1,p2) -> instead, where p1 and p2 are parameters. Let’s make our hello_world function say our name: #COFFEESCRIPT hello_world = (name) -> console.log “Hello #{name}” //JAVSCRIPT – OUTPUT varhello_world; hello_world = function(name) { returnconsole.log(“Hello “ + name); } In this example, we see how the special word function disappears and string interpolation. CoffeeScript allows the programmer to easily add logic to a string by escaping the string with #{}. Unlike JavaScript, you can also add returns and reshape the way astring looks without breaking the code. Binding In Meteor, we will often find ourselves using the properties of bindingwithin nested functions and callbacks.Function binding is very useful for these types of cases and helps avoid having to save data in additional variables. Let’s look at an example: #COFFEESCRIPT # Let’s make the context of this equal to our toolbox object # this = # hammer:true # flashlight:false # Run a method with a callback Meteor.call “use_hammer”, -> console.log this In this case, the thisobjectwill return a top-level object, such as the browser window. That's not useful at all. Let’s bind it now: #COFFEESCRIPT # Let’s make the context of this equal to our toolbox object # this = # hammer:true # flashlight:false # Run a method with a callback Meteor.call “use_hammer”, => console.log this The key difference is the use of =>instead of the expected ->sign fordefining the function. This will ensure that the callback'sthis object contains the context of the executing function. The resulting compiled script is as follows: //JAVASCRIPT Meteor.call(“use_hammer”, (function(_this) { return function() { returnConsole.log(_this); }; })(this)); CoffeeScript will improve your code and help you write codefaster. Still, itis not flawless. When you start combining functions with nested arrays, things can get complex and difficult to read, especially when functions are constructed with multiple parameters. Let’s look at an ugly query: #COFFEESCRIPT People.update sibling: $in:[“bob”,”bill”] , limit:1 -> console.log “success!” There are a few ways ofexpressing the difference between two different parameters of a function, but by far the easiest to understand. We place a comma one indentation before the next object. Go to coffeescript.org and play around with the language by clicking on the try coffeescript link. Summary We can now program faster because we have tools such as CoffeeScript, Jade, and Stylus to help us. We also seehow to use templates, helpers, and events to make our frontend work with Meteor. Resources for Article: Further resources on this subject: Why Meteor Rocks! [article] Function passing [article] Meteor.js JavaScript Framework: Why Meteor Rocks! [article]

0
0
6788

Packt

14 May 2010

12 min read

Personalizing Vim

Packt

14 May 2010

12 min read

0
0
6786

Packt

16 Dec 2014

9 min read

Ridge Regression

Packt

16 Dec 2014

9 min read

In this article by Patrick R. Nicolas, the author of the book Scala for Machine Learning, we will cover the basics of ridge regression. The purpose of regression is to minimize a loss function, the residual sum of squares (RSS) being the one commonly used. The problem of overfitting can be addressed by adding a penalty term to the loss function. The penalty term is an element of the larger concept of regularization. (For more resources related to this topic, see here.) Ln roughness penalty Regularization consists of adding a penalty function J(w) to the loss function (or RSS in the case of a regressive classifier) in order to prevent the model parameters (or weights) from reaching high values. A model that fits a training set very well tends to have many features variable with relatively large weights. This process is known as shrinkage. Practically, shrinkage consists of adding a function with model parameters as an argument to the loss function: The penalty function is completely independent from the training set {x,y}. The penalty term is usually expressed as a power to function of the norm of the model parameters (or weights) wd. For a model of D dimension the generic Lp-norm is defined as follows: Notation Regularization applies to parameters or weights associated to an observation. In order to be consistent with our notation w0 being the intercept value, the regularization applies to the parameters w1 …wd. The two most commonly used penalty functions for regularization are L1 and L2. Regularization in machine learning The regularization technique is not specific to the linear or logistic regression. Any algorithm that minimizes the residual sum of squares, such as support vector machine or feed-forward neural network, can be regularized by adding a roughness penalty function to the RSS. The L1 regularization applied to the linear regression is known as the Lasso regularization. The Ridge regression is a linear regression that uses the L2 regularization penalty. You may wonder which regularization makes sense for a given training set. In a nutshell, L2 and L1 regularizations differ in terms of computation efficiency, estimation, and features selection (refer to the 13.3 L1 regularization: basics section in the book Machine Learning: A Probabilistic Perspective, and the Feature selection, L1 vs. L2 regularization, and rotational invariance paper available at http://www.machinelearning.org/proceedings/icml2004/papers/354.pdf). The various differences between the two regularizations are as follows: Model estimation: L1 generates a sparser estimation of the regression parameters than L2. For large non-sparse dataset, L2 has a smaller estimation error than L1. Feature selection: L1 is more effective in reducing the regression weights for features with high value than L2. Therefore, L1 is a reliable features selection tool. Overfitting: Both L1 and L2 reduce the impact of overfitting. However, L1 has a significant advantage in overcoming overfitting (or excessive complexity of a model) for the same reason it is more appropriate for selecting features. Computation: L2 is conducive to a more efficient computation model. The summation of the loss function and L2 penalty w2 is a continuous and differentiable function for which the first and second derivative can be computed (convex minimization). The L1 term is the summation of |wi|, and therefore, not differentiable. Terminology The ridge regression is sometimes called the penalized least squares regression. The L2 regularization is also known as the weight decay. Let's implement the ridge regression, and then evaluate the impact of the L2-norm penalty factor. Ridge regression The ridge regression is a multivariate linear regression with a L2 norm penalty term, and can be calculated as follows: The computation of the ridge regression parameters requires the resolution of the system of linear equations similar to the linear regression. Matrix representation of ridge regression closed form is as follows: I is the identity matrix and it is using the QR decomposition, as shown here: Implementation The implementation of the ridge regression adds L2 regularization term to the multiple linear regression computation of the Apache Commons Math library. The methods of RidgeRegression have the same signature as its ordinary least squares counterpart. However, the class has to inherit the abstract base class AbstractMultipleLinearRegression in the Apache Commons Math and override the generation of the QR decomposition to include the penalty term, as shown in the following code: class RidgeRegression[T <% Double](val xt: XTSeries[Array[T]], val y: DblVector, val lambda: Double) { extends AbstractMultipleLinearRegression with PipeOperator[Array[T], Double] { private var qr: QRDecomposition = null private[this] val model: Option[RegressionModel] = … … } Besides the input time series xt and the labels y, the ridge regression requires the lambda factor of the L2 penalty term. The instantiation of the class train the model. The steps to create the ridge regression models are as follows: Extract the Q and R matrices for the input values, newXSampleData (line 1) Compute the weights using the calculateBeta defined in the base class (line 2) Return the tuple regression weights calculateBeta and the residuals calculateResiduals private val model: Option[(DblVector, Double)] = { this.newXSampleData(xt.toDblMatrix) //1 newYSampleData(y) val _rss = calculateResiduals.toArray.map(x => x*x).sum val wRss = (calculateBeta.toArray, _rss) //2 Some(RegressionModel(wRss._1, wRss._2)) } The QR decomposition in the AbstractMultipleLinearRegression base class does not include the penalty term (line 3); the identity matrix with lambda factor in the diagonal has to be added to the matrix to be decomposed (line 4). override protected def newXSampleData(x: DblMatrix): Unit = { super.newXSampleData(x) //3 val xtx: RealMatrix = getX val nFeatures = xt(0).size Range(0, nFeatures).foreach(i => xtx.setEntry(i,i,xtx.getEntry(i,i) + lambda)) //4 qr = new QRDecomposition(xtx) } The regression weights are computed by resolving the system of linear equations using substitution on the QR matrices. It overrides the calculateBeta function from the base class: override protected def calculateBeta: RealVector = qr.getSolver().solve(getY()) Test case The objective of the test case is to identify the impact of the L2 penalization on the RSS value, and then compare the predicted values with original values. Let's consider the first test case related to the regression on the daily price variation of the Copper ETF (symbol: CU) using the stock daily volatility and volume as feature. The implementation of the extraction of observations is identical as with the least squares regression: val src = DataSource(path, true, true, 1) val price = src |> YahooFinancials.adjClose val volatility = src |> YahooFinancials.volatility val volume = src |> YahooFinancials.volume //1 val _price = price.get.toArray val deltaPrice = XTSeries[Double](_price .drop(1) .zip(_price.take(_price.size -1)) .map( z => z._1 - z._2)) //2 val data = volatility.get .zip(volume.get) .map(z => Array[Double](z._1, z._2)) //3 val features = XTSeries[DblVector](data.take(data.size-1)) val regression = new RidgeRegression[Double](features, deltaPrice, lambda) //4 regression.rss match { case Some(rss) => Display.show(rss, logger) …. The observed data, ETF daily price, and the features (volatility and volume) are extracted from the source src (line 1). The daily price change, deltaPrice, is computed using a combination of Scala take and drop methods (line 2). The features vector is created by zipping volatility and volume (line 3). The model is created by instantiating the RidgeRegression class (line 4). The RSS value, rss, is finally displayed (line 5). The RSS value, rss, is plotted for different values of lambda <= 1.0 in the following graph: Graph of RSS versus Lambda for Copper ETF The residual sum of squares decreased as λ increases. The curve seems to be reaching for a minimum around λ=1. The case of λ = 0 corresponds to the least squares regression. Next, let's plot the RSS value for λ varying between 1 and 100: Graph RSS versus large value Lambda for Copper ETF This time around RSS increases with λ before reaching a maximum for λ > 60. This behavior is consistent with other findings (refer to Lecture 5: Model selection and assessment, a lecture by H. Bravo and R. Irizarry from department of Computer Science, University of Maryland, in 2010, available at http://www.cbcb.umd.edu/~hcorrada/PracticalML/pdf/lectures/selection.pdf). As λ increases, the overfitting gets more expensive, and therefore, the RSS value increases. The regression weights can by simply outputted as follows: regression.weights.get Let's plot the predicted price variation of the Copper ETF using the ridge regression with different value of lambda (λ): Graph of ridge regression on Copper ETF price variation with variable Lambda The original price variation of the Copper ETF Δ = price(t+1)-price(t) is plotted as λ =0. The predicted values for λ = 0.8 is very similar to the original data. The predicted values for λ = 0.8 follows the pattern of the original data with reduction of large variations (peaks and troves). The predicted values for λ = 5 corresponds to a smoothed dataset. The pattern of the original data is preserved but the magnitude of the price variation is significantly reduced. The reader is invited to apply the more elaborate K-fold validation routine and compute precision, recall, and F1 measure to confirm the findings. Summary The ridge regression is a powerful alternative to the more common least squares regression because it reduces the risk of overfitting. Contrary to the Naïve Bayes classifiers, it does not require conditional independence of the model features. Resources for Article: Further resources on this subject: Differences in style between Java and Scala code [Article] Dependency Management in SBT [Article] Introduction to MapReduce [Article]

0
0
6786

article-image-ext-js-4-working-grid-component

Packt

11 Jan 2012

10 min read

Ext JS 4: Working with the Grid Component

Packt

11 Jan 2012

10 min read

0
0
6783

Packt

04 Jan 2016

12 min read

Assessment Planning

Packt

04 Jan 2016

12 min read

In this article by Kevin Cardwell the author of the book Advanced Penetration Testing for Highly-Secured Environments - Second Edition, discusses the test environment and how we have selected the chosen platform. We will discuss the following: Introduction to advanced penetration testing How to successfully scope your testing (For more resources related to this topic, see here.) Introduction to advanced penetration testing Penetration testing is necessary to determine the true attack footprint of your environment. It may often be confused with vulnerability assessment and thus it is important that the differences should be fully explained to your clients. Vulnerability assessments Vulnerability assessments are necessary for discovering potential vulnerabilities throughout the environment. There are many tools available that automate this process so that even an inexperienced security professional or administrator can effectively determine the security posture of their environment. Depending on scope, additional manual testing may also be required. Full exploitation of systems and services is not generally in scope for a normal vulnerability assessment engagement. Systems are typically enumerated and evaluated for vulnerabilities, and testing can often be done with or without authentication. Most vulnerability management and scanning solutions provide actionable reports that detail mitigation strategies such as applying missing patches, or correcting insecure system configurations. Penetration testing Penetration testing expands upon vulnerability assessment efforts by introducing exploitation into the mix The risk of accidentally causing an unintentional denial of service or other outage is moderately higher when conducting a penetration test than it is when conducting vulnerability assessments. To an extent, this can be mitigated by proper planning, and a solid understanding of the technologies involved during the testing process. Thus, it is important that the penetration tester continually updates and refines the necessary skills. Penetration testing allows the business to understand if the mitigation strategies employed are actually working as expected; it essentially takes the guesswork out of the equation. The penetration tester will be expected to emulate the actions that an attacker would attempt and will be challenged with proving that they were able to compromise the critical systems targeted. The most successful penetration tests result in the penetration tester being able to prove without a doubt that the vulnerabilities that are found will lead to a significant loss of revenue unless properly addressed. Think of the impact that you would have if you could prove to the client that practically anyone in the world has easy access to their most confidential information! Penetration testing requires a higher skill level than is needed for vulnerability analysis. This generally means that the price of a penetration test will be much higher than that of a vulnerability analysis. If you are unable to penetrate the network you will be ensuring your clientele that their systems are secure to the best of your knowledge. If you want to be able to sleep soundly at night, I recommend that you go above and beyond in verifying the security of your clients. Advanced penetration testing Some environments will be more secured than others. You will be faced with environments that use: Effective patch management procedures Managed system configuration hardening policies Multi-layered DMZ's Centralized security log management Host-based security controls Network intrusion detection or prevention systems Wireless intrusion detection or prevention systems Web application intrusion detection or prevention systems Effective use of these controls increases the difficulty level of a penetration test significantly. Clients need to have complete confidence that these security mechanisms and procedures are able to protect the integrity, confidentiality, and availability of their systems. They also need to understand that at times the reason an attacker is able to compromise a system is due to configuration errors, or poorly designed IT architecture. Note that there is no such thing as a panacea in security. As penetration testers, it is our duty to look at all angles of the problem and make the client aware of anything that allows an attacker to adversely affect their business. Advanced penetration testing goes above and beyond standard penetration testing by taking advantage of the latest security research and exploitation methods available. The goal should be to prove that sensitive data and systems are protected even from a targeted attack, and if that is not the case, to ensure that the client is provided with the proper instruction on what needs to be changed to make it so. A penetration test is a snapshot of the current security posture. Penetration testing should be performed on a continual basis. Many exploitation methods are poorly documented, frequently hard to use, and require hands-on experience to effectively and efficiently execute. At DefCon 19 Bruce "Grymoire" Barnett provided an excellent presentation on "Deceptive Hacking". In this presentation, he discussed how hackers use many of the very same techniques used by magicians. This is exactly the tenacity that penetration testers must assume as well. Only through dedication, effort, practice, and the willingness to explore unknown areas will penetration testers be able to mimic the targeted attack types that a malicious hacker would attempt in the wild. Often times you will be required to work on these penetration tests as part of a team and will need to know how to use the tools that are available to make this process more endurable and efficient. This is yet another challenge presented to today's pentesters. Working in a silo is just not an option when your scope restricts you to a very limited testing period. In some situations, companies may use non-standard methods of securing their data, which makes your job even more difficult. The complexity of their security systems working in tandem with each other may actually be the weakest link in their security strategy. The likelihood of finding exploitable vulnerabilities is directly proportional to the complexity of the environment being tested. Before testing begins Before we commence with testing, there are requirements that must be taken into consideration. You will need to determine the proper scoping of the test, timeframes and restrictions, the type of testing (Whitebox, Blackbox), and how to deal with third-party equipment and IP space. Determining scope Before you can accurately determine the scope of the test, you will need to gather as much information as possible. It is critical that the following is fully understood prior to starting testing procedures: Who has the authority to authorize testing? What is the purpose of the test? What is the proposed timeframe for the testing? Are there any restrictions as to when the testing can be performed? Does your customer understand the difference between a vulnerability assessment and a penetration test? Will you be conducting this test with, or without cooperation of the IT Security Operations Team? Are you testing their effectiveness? Is social engineering permitted? How about denial-of-service attacks? Are you able to test physical security measures used to secure servers, critical data storage, or anything else that requires physical access? For example, lock picking, impersonating an employee to gain entry into a building, or just generally walking into areas that the average unaffiliated person should not have access to. Are you allowed to see the network documentation or to be informed of the network architecture prior to testing to speed things along? (Not necessarily recommended as this may instill doubt for the value of your findings. Most businesses do not expect this to be easy information to determine on your own.) What are the IP ranges that you are allowed to test against? There are laws against scanning and testing systems without proper permissions. Be extremely diligent when ensuring that these devices and ranges actually belong to your client or you may be in danger of facing legal ramifications. What are the physical locations of the company? This is more valuable to you as a tester if social engineering is permitted because it ensures that you are at the sanctioned buildings when testing. If time permits, you should let your clients know if you were able to access any of this information publicly in case they were under the impression that their locations were secret or difficult to find. What to do if there is a problem or if the initial goal of the test has been reached. Will you continue to test to find more entries or is the testing over? This part is critical and ties into the question of why the customer wants a penetration test in the first place. Are there legal implications that you need to be aware of such as systems that are in different countries, and so on? Not all countries have the same laws when it comes to penetration testing. Will additional permission be required once a vulnerability has been exploited? This is important when performing tests on segmented networks. The client may not be aware that you can use internal systems as pivot points to delve deeper within their network. How are databases to be handled? Are you allowed to add records, users, and so on? This listing is not all-inclusive and you may need to add items to the list depending on the requirements of your clients. Much of this data can be gathered directly from the client, but some will have to be handled by your team. If there are legal concerns, it is recommended that you seek legal counsel to ensure you fully understand the implications of your testing. It is better to have too much information than not enough, once the time comes to begin testing. In any case, you should always verify for yourself that the information you have been given is accurate. You do not want to find out that the systems you have been accessing do not actually fall under the authority of the client! It is of utmost importance to gain proper authorization in writing before accessing any of your clients systems. Failure to do so may result in legal action and possibly jail. Use proper judgment! You should also consider that errors and omissions insurance is a necessity when performing penetration testing. Setting limits–nothing lasts forever Setting proper limitations is essential if you want to be successful at performing penetration testing. Your clients need to understand the full ramifications involved, and should be made aware of any residual costs incurred, if additional services beyond those listed within the contract are needed. Be sure to set defined start and end dates for your services. Clearly define the rules of engagement and include IP ranges, buildings, hours, and so on that may need to be tested. If it is not in your rules of engagement documentation, it should not be tested. Meetings should be predefined prior to the start of testing, and the customer should know exactly what your deliverables will be. Rules of engagement documentation Every penetration test will need to start with a rules of engagement document that all involved parties must have. This document should at a minimum cover several items: Proper permissions by appropriate personnel. Begin and end dates for your testing. The type of testing that will be performed. Limitations of testing. What type of testing is permitted? DDOS? Full penetration? Social engineering? These questions need to be addressed in detail. Can intrusive tests as well as unobtrusive testing be performed? Does your client expect cleanup to be performed afterwards or is this a stage environment that will be completely rebuilt after testing has been completed? IP ranges and physical locations to be tested. How the report will be transmitted at the end of the test? (Use secure means of transmission!) Which tools will be used during the test? Do not limit yourself to only one specific tool; it may be beneficial to provide a list of the primary toolset to avoid confusion in the future. For example, we will use the tools found in the most recent edition of the Kali Suite. Let your client know how any illegal data that is found during testing would be handled: law enforcement should be contacted prior to the client. Please be sure to understand fully the laws in this regard before conducting your test. How sensitive information will be handled: you should not be downloading sensitive customer information; there are other methods of proving that the clients' data is not secured. This is especially important when regulated data is a concern. Important contact information for both your team and for the key employees of the company you are testing. An agreement of what you will do to ensure the customer's system information does not remain on unsecured laptops and desktops used during testing. Will you need to properly scrub your machine after this testing? What do you plan to do with the information you gathered? Is it to be kept somewhere for future testing? Make sure this has been addressed before you start testing, not after. The rules of engagement should contain all the details that are needed to determine the scope of the assessment. Any questions should have been answered prior to drafting your rules of engagement to ensure there are no misunderstandings once the time comes to test. Your team members need to keep a copy of this signed document on their person at all times when performing the test. Imagine you have been hired to assert the security posture of a client's wireless network and you are stealthily creeping along the parking lot on private property with your gigantic directional Wi-Fi antenna and a laptop. If someone witnesses you in this act, they will probably be concerned and call the authorities. You will need to have something on you that documents you have a legitimate reason to be there. This is one time where having the contact information of the business leaders that hired you will come in extremely handy! Summary In this article, we focused on all that is necessary to prepare and plan for a successful penetration test. We discussed the differences between penetration testing and vulnerability assessments. The steps involved with proper scoping were detailed, as were the necessary steps to ensure all information has been gathered prior to testing. One thing to remember is that proper scoping and planning is just as important as ensuring you test against the latest and greatest vulnerabilities. Resources for Article: Further resources on this subject: Penetration Testing[article] Penetration Testing and Setup[article] BackTrack 4: Security with Penetration Testing Methodology[article]

0
0
6781

How-To Tutorials

article-image-microsoft-dynamics-gp-data-management

Packt

27 Apr 2011

13 min read

Microsoft Dynamics GP: Data Management

Packt

27 Apr 2011

13 min read

Microsoft Dynamics GP 2010 Reporting Create and manage business reports with Dynamics GP Knowing where to begin is a critical first step in the development process. The aim of this article is to provide helpful tips for finding and locating data in the Dynamics GP 2010 ERP system and company databases. Although we'll discuss some reporting tools that do not require us to know the SQL database structure for Dynamics GP companies, it is still helpful to understand how GP stores its data. DYNAMICS database versus company database The first task of identifying where our data is located is making sure we are using the correct database! Microsoft Dynamics GP 2010 utilizes the Microsoft SQL Server platform as its database engine. When Dynamics GP 2010 is installed on our environment and a new company is created, the installation process creates several databases on a server that has been designated during the install. These databases will store all the information entered through the Dynamics GP 2010 application, and we can use SQL Server Management Studio to access the underlying tables that store this data. Microsoft Dynamics GP has two types of databases: a system database (DYNAMICS) and company database(s). For first-time report developers and seasoned writers, knowing which of these databases stores a particular piece of information we need is crucial for an accurate report. What is the DYNAMICS database? When Microsoft Dynamics GP is first installed and some initial settings are provided, a system database will be created. This database is the DYNAMICS database. It includes such things as records for each company that you create, the organizations registration information, the maximum account framework, and so on. Regardless of how many company databases we have, they will all share a single DYNAMICS database. From a reporting standpoint, there is certain information located in the system database that we may need to report on at one time or another. We have provided a quick reference for this information below: Multicurrency System Setups – This includes the setup of the currencies, the exchange rates, and the currency symbols for those organizations that process transactions in any number of foreign currencies. Intercompany Setup – This is where the intercompany relationships are stored and the specific due to/due from accounts are mapped. Organizations Structures – This will include the organizational levels and entities that have been created for those organizations that use this feature. User Master – This table stores information about the users in the ERP system, including their user ID and username. User Tasks – This table stores the tasks that users set up in Dynamics GP. These are the tasks that are displayed on the users' Home screens in GP. Company Master – Stores company setup information such as whether security is enabled, the company ID is in the form of the company database ID in SQL Management Studio, primary address information, tax schedule defaults, and any number of options for the company. Security Setups – This includes all of the security tasks, security roles, and user security assignments. User-Company Access – Includes the companies that each user has access to. Company database Each company that we create in Dynamics GP has its own company database. As information such as transactions, accounts, and customer or vendor data is entered through GP, this information is recorded in individual fields. These fields comprise the smallest unit of data stored. All of this data makes up a record and a record is grouped with similar records and stored in a table. For obvious reasons, this data is segmented by company database so that each company can maintain unique records. In addition to this transactional data, numerous additional company setups exist in the company database. As with the DYNAMICS database, we may need to report on some of these company system setups. Here is a quick reference to the more common company setup tables: Account Formats – Stores the chart of accounts format for the company. Posting Definitions – Stores how the individual modules post to the General Ledger. Company Locations – Lists additional addresses for each company. Source Document Master & Audit Trail Codes – Every transaction is assigned both a source document and an audit trail code. This table can be used to report on the full description of these codes. Shipping Methods Master – Stores the setup details of the shipping methods for the company. Payment Terms Master – Stores the setup details of the payment terms created for the company. Record Notes Master – Stores all of the record level notes for the particular company. Comment Master – Stores predefined comments to be used across multiple series in Dynamics GP. Electronic Funds Transfer – Stores the EFT setup information for the company for both Payables and Receivables modules. This includes customer and vendor banking information. Period Setup – Stores the fiscal period setup for the company. Sales/Purchases Tax Tables – These tables store the tax detail and tax schedule records as well as the tax summary amounts. Dynamics GP table naming/numbering conventions Dynamics GP has a rather interesting and sometimes frustrating table naming structure. When developers or consultants first see this, they are overwhelmed, to say the least. However, once you learn that there is a rhyme and reason to the madness, it actually makes a lot of sense and it is quite easy to follow and locate the tables that we need. Tables versus table groups When data is entered into windows via the Microsoft Dynamics GP application, that data is stored in tables in the underlying SQL database. In most cases, data entered via a single process can be stored in two or more tables. In such cases, it is common for these tables to be grouped together by a certain naming convention. For example, entering journal entry information may update the Transactions Work table (contains General Ledger transaction header information), the Transaction Amounts Work table (contains the General Ledger transaction distributions), and the Transaction Clearing Amounts Work table (contains the General Ledger clearing transactions distributions). These make up what are called Table Groups. These table groups are also referred to as logical tables. Each Microsoft Dynamics GP table has three names: a technical name, a display name, and a physical name. The technical name is used solely by the software and will usually be seen in some alert messages instead of the display name. The display name is the name that will appear in most of the alert messages generated by the system and is typically the name used for a given table when referring to it in speech, for example, Vendor Master. The physical name is the name that will be found in the SQL database when looking in Microsoft SQL Server Management Studio. Physical table naming/numbering conventions When working within the context of a SQL database to generate reports, developers and consultants will make use of the table physical names. A quick scan through the various tables in a standard GP install reveals a bewildering array of table numbers. How can there possibly be any rhyme or reason to these table names? Surprisingly, it does actually follow a certain pattern. For the most part, the table numbering follows a special convention. This schema packs a lot of information into a small number, and it can help developers and consultants know where to begin looking for their data. As Dynamics GP has grown into a more comprehensive accounting solution, it has expanded, in part, by incorporating third-party applications into the solution. While many of the third-party programmers tried to stick within the relative bounds of the GP physical table naming conventions, as we will soon see, this is not always the case. So, while many of the tables that belong to the core GP modules maintain a fairly standard numbering convention, we will find that this does not hold true for all GP modules. Generally speaking, GP table physical names contain a two or three digit alpha prefix followed by a five digit number. The three digit prefix represents the module for which the table holds data. The numbers that follow identify what type of data is held in the table. For example, is it posted transaction data? Or, is it information related to the module setup? As we see in the next image and following sections of this chapter, the numbering schema for a Dynamics GP physical table can be broken down to reveal information about the kind of data that is found in that table. Alpha code Let's begin by taking a look at the alpha-prefix for these tables. We have put together a handy reference of some of the most common prefixes and the modules that they represent: PrefixModulePrefixModuleAAAnalytical AccountingMRPMaterial Requirements PlanningAFAdvanced FinancialsMXLSAudit Trails/Electronic SignaturesASISmartListNLBNavigation List BuilderBMBill of Materials (Mfg)OCSales Configurator (Mfg)CFMCash Flow ManagementOSRCOutsourcing (Mfg)CLMCertification ManagerPAProject AccountingCMCheckbook MasterPDKPersonal Data Keeper (Proj. Acct.)CNCollections ManagementPMPayables ManagementCPCapacity Requirements Planning (Mfg)POPPurchase Order ProcessingDDDirect DepositPPRevenue Expense DeferralsECEngineering Change Management (Mfg)QAQuality Assurance (Mfg)ERBExcel Report BuilderRMReceivables ManagementEXTExtenderRTRoutings (Mfg)FAFixed AssetsSCSales Forecasting (Mfg)GLGeneral LedgerSLBSmartList BuilderHRHuman ResourcesSOPSales Order ProcessingICJCJob Costing (Mfg)SVCField ServiceIVInventorySYSystem/Company SetupIVCInvoicing (Sales)UPRPayrollMCMulticurrencyVATIntrastatMEElectronic Reconcile (EFT)WCWork Centers (Mfg)MOPManufacturing Order ProcessingWOManufacturing Orders In some modules, such as Manufacturing, tables are broken down even further with Routing tables represented by one set of digits while Material Requirements Planning data is found in tables represented by another set of digits. More commonly, however, entire modules are generally represented by a single alpha code, such as is the case with Project Accounting where all project transactions, billing, and revenue recognition tables are represented with the same alpha code. Table type Once we've identified the module in which our data might be stored, our next thought should be towards the type of data that we are trying to find. Before we return to the numbering convention, let's take a look at the various types of data that exist in GP tables: Setup: Almost all modules in GP have set up windows that allow users to define default options or other settings for how that module will be used. The options selected in these windows can be found in the Setup tables. Master: Some modules, such as Payables Management, allow users to record master records. These master records represent permanent records, such as vendors, for the company. Typically, master records must be entered prior to using a module, as transactions will utilize these master records. Transaction: These tables contain the transaction-level data from Dynamics GP. These include the most basic of GP transactions, the journal entry, to transactions entered in sub-ledgers, such as the distribution records of a posted Receivables invoice. Transactions entered in various modules will, depending on their status, be stored in one of three types of transaction tables: Work: Un-posted transactions can generally be found in work tables. The name is appropriate, as these transactions can be considered a "work-in-progress". They have not been committed to the sub or general ledgers via posting processes, so we should factor this into our thinking when deciding whether or not to include these records in our reports. Open: Generally speaking, records in these tables have been posted, but are awaiting an additional action before they can be considered "closed" or "history". In the General Ledger module, the open table represents all journal entries for the current open year. In other modules, such as Receivables Management, open tables contain data for receivables transactions that have not yet been fully applied. History: Transactions that are "completed" generally end up in the history tables. Again, this depends on the module. For example, in Payables Management, fully applied invoices are moved to history. In Purchase Order Processing, however, a routine exists to move completed purchase orders to history. Until this routine is run, records will not be moved to history. Cross Reference: Some tables represent data that spans multiple modules. For example, GP users can link purchase orders to unfulfilled sales order line items via Sales Order Commitment. The prefix of the table that contains these links indicates that this is a Sales Order Processing table, but in actuality, it contains data from Purchase Order Processing, as well. In addition to these main table types, other table types exist that are less commonly used for reporting purposes. Nonetheless, it is helpful to understand what these table types contain: Report Options: Before users can generate a report from the Reports menu, a series of options must be designated for that report. These report options are recorded in a series of report options tables. Temp: As the name implies, data is only stored in these tables temporarily. Temporary tables can be used in a variety of situations, such as when a user presses the "Post" button for a transaction. Although rare, it may be necessary to use a temp table when designing a report that should contain data from the time of posting. For example, a sales invoice can be generated at the time the user posts the invoice and can be based on data stored in a temp table at the time. In terms of the physical naming convention for GP tables, the data type is represented by the first digit following the module code. The following table contains the various table types and their associated number in the numbering convention: Table TypeNumberMaster0Work1Open2History3Setup4Temporary5Cross Reference6Report Options7 Sequence The next two digits in the numbering convention make up the sequence number. This number indicates the logical table to which the table belongs. As we discussed earlier in the article, logical tables are related tables, or table groups, that share similar data. Not only do these table groups share similar data, but they also share the same data type and sequence number when it comes to the physical table numbering convention. For example, let's consider the following set of logical tables: PM00200 (Vendor Master) PM00201 (Vendor Master Summary) PM00202 (Vendor Master Period Summary) PM00203 (Vendor Accounts) PM00204 (Purchasing 1099 Detail) These tables comprise the Payables Vendor Master Logical File table group. We can easily see this by the numbers that follow the module code. First, we see that these tables share the same data type—remember, 0 means these are Master tables—and second, we see that these tables share the same sequence number. Although we need other tools to help us determine the name of the table group, we are able to easily scan through a list of table physical names in SQL Management Studio and see that these tables are in the same table group. Variant The final two digits of the physical numbering convention represent the logical group variant. Within a logical group, numbers are incremented sequentially. This is evident in our example using the Payables Vendor Master Logical File above; as we see the final two digits of each table increment by one. In table groups related to transactions, the variant often distinguishes between header tables, detail tables, distribution tables, and other related tables. Keep in mind, what we have discussed is only a general naming and numbering convention for GP tables in their SQL databases. Unfortunately, to the frustration of developers and consultants everywhere, these conventions do not hold true in all cases! At the very least, knowing this convention can point us in the right direction. We can rely on other tools and resources to help us pinpoint the right table when these conventions fall short.

0
0
6779

How-To Tutorials

article-image-introduction-editing-operators-blender

Packt

29 Mar 2011

6 min read

Introduction to the Editing Operators in Blender

Packt

29 Mar 2011

6 min read

Blender 2.5 Materials and Textures Cookbook Over 80 great recipes to create life-like Blender objects For editing task, 3D View has some features to help you in your work. These commands are easily executable from the Object/Mesh operators panel (T Key with mouse cursor over 3D View Editor). Here you have some different options depending if you are in Object Mode or Edit Mode. You can work according to different Modes in Blender (Object Mode, Edit Mode, Sculpt Mode, Pose Mode, etc) and accordingly you will be able to deal with different properties and operators. For modeling, you will use Edit Mode and Object Mode. (See illustration 1) Illustration 1: The Mode Menu Changing Mode in Blender modifies your workflow and offers you different options/properties according to the chosen mode. Differences between Object and Edit Mode To understand the differences between Object and Edit Mode easily, I'll make a comparison. In Blender, an Object is the physical representation of anything, and Mesh is the material that it is made up of. If you have got a plasticine puppet over the table, and if you see it, move it, and bring it to trash, then your puppet is in Object Mode. If you want to modify it by stretching its legs, adding more plasticine or something similar, then you need to manipulate it. This action is done over the Mesh in Edit Mode in Blender. It is important that you get this idea to understand at every moment the Mode you are working on. Then, you are in 3D View Editor and Object Mode (by default). If you got the default user scene with a cube, just select the cube (selected by default) with RMB (Right Mouse Button) and delete it with X Key. The purpose of this article is not to create the whole model, but to introduce to you the operators used for creating it. I will begin by telling you how to start a single model. Practical Exercise: Go to the Top view in 3D View Editor by pressing 7 Key in the Numpad. Take care you are not in the Persp view. You could check in the top left corner of the editor. If you are in the “Top Persp” then press 5 Key in the Numpad to go to “Top Ortho”. If you were there from the start, just avoid this step. We'll add our first plane to the scene. This plane will become our main character at the end. To add this plane just press Shift+A or click on the Add Plane| in the top menu in the Info Editor. (See illustration 2 and 3) Illustration 2: Add Menu. Floating The Add Object Menu is accessible within 3D View Editor at any moment by pressing Shift+A, resulting in a floating menu. Illustration 3: Add Menu The Add Object Menu is also accessible from the Add option within the Info Editor (at the top by default). Once we select Plane from that menu, we have a plane in our scene, 3D View Editor and Top View. This plane is currently in Object Mode. You can check this in the Mode Menu we have seen before. Now go to Right View by pressing 3 Key in the Numpad, then press R Key to rotate the plane and enter -90. Press Enter after that. Then go to the Front View by pressing 1 Key in the Numpad. We are now going to apply some very interesting Modifiers to the plane to help us in the modeling stage. We have our first plane in vertical position in Front View and in Object Mode. Press Tab Key to enter the Edit Mode and W Key to Subdivide the plane. A floating menu appears after pressing W Key. This is the Specials Menu with very interesting options there. At the moment Subdivide is the one we are interested in. (See illustration 4) Illustration 4: Specials Menu Specials Menu helps you in the Editing stage with interesting operators like Subdivide, Merge, Remove Doubles, etc. We will use some of these operators in future steps, so keep an eye on this panel. Well, back to the model. We have our plane in 3D View Editor, Edit Mode and Subdivide was recently applied. Notice that you will have a plane subdivided into four little planes by default. The amount of subdivision can be modified in the Subdivide panel below Mesh operators (If collapsed, T Key with mouse cursor over 3D View Editor) as we have seen previously. You now have a Subdivide Number of Cuts. To keep the target, we will go with the default one. So then we have a plane subdivided in little ones (four). By default all vertices are selected, to deselect all vertices, press A Key. Now our plane has no vertex selected. Next step is to set up a Mirror Modifier to help us model just the left mid (right mid for us because we are looking at our model in Front View). With all vertices deselected, press B Key to activate Border Select and make a square selecting left side of the plane with LMB (Left Mouse Button). (See illustration 5) Illustration 5: Border Select To select or deselect vertex, edges or faces in Edit Mode you could use Border Select (B Key) or Circle Select (C Key). First you need to make a square to select. Second is a circle that can be selected by clicking and dragging like in painting editors. Diameter of Circle Select could be adjusted with MMB (Middle Mouse Button). With left side of the plane selected, press X Key to delete those vertices. The Delete Menu will then offer to you different options to delete, in our case just vertex option. We now have only the right side of the panel, to which we will apply the Mirror Modifier. For that we must know at this point where to find them. To manage different operators or actions in our objects or meshes, we have the Buttons that will open the right Property Editor, according to our needs. (See illustration 6) Illustration 6: Buttons Selector Buttons open the right Property Editor.

0
0
6777

article-image-getting-started-pentaho-data-integration

Packt

30 Oct 2013

16 min read

Getting Started with Pentaho Data Integration

Packt

30 Oct 2013

16 min read

(For more resources related to this topic, see here.) Pentaho Data Integration and Pentaho BI Suite Before introducing PDI, let’s talk about Pentaho BI Suite. The Pentaho Business Intelligence Suite is a collection of software applications intended to create and deliver solutions for decision making. The main functional areas covered by the suite are: Analysis: The analysis engine serves multidimensional analysis. It’s provided by the Mondrian OLAP server. Reporting: The reporting engine allows designing, creating, and distributing reports in various known formats (HTML, PDF, and so on), from different kinds of sources. Data Mining: Data mining is used for running data through algorithms in order to understand the business and do predictive analysis. Data mining is possible thanks to the Weka Project. Dashboards: Dashboards are used to monitor and analyze Key Performance Indicators (KPIs). The Community Dashboard Framework (CDF), a plugin developed by the community and integrated in the Pentaho BI Suite, allows the creation of interesting dashboards including charts, reports, analysis views, and other Pentaho content, without much effort. Data Integration: Data integration is used to integrate scattered information from different sources (applications, databases, files, and so on), and make the integrated information available to the final user. All of this functionality can be used standalone but also integrated. In order to run analysis, reports, and so on, integrated as a suite, you have to use the Pentaho BI Platform. The platform has a solution engine, and offers critical services, for example, authentication, scheduling, security, and web services. This set of software and services form a complete BI Platform, which makes Pentaho Suite the world’s leading open source Business Intelligence Suite. Exploring the Pentaho Demo The Pentaho BI Platform Demo is a pre-configured installation that allows you to explore several capabilities of the Pentaho platform. It includes sample reports, cubes, and dashboards for Steel Wheels. Steel Wheels is a fictional store that sells all kind of scale replicas of vehicles. The following screenshot is a sample dashboard available in the demo: The Pentaho BI Platform Demo is free and can be downloaded from http://sourceforge.net/projects/pentaho/files/. Under the Business Intelligence Server folder, look for the latest stable version. You can find out more about Pentaho BI Suite Community Edition at http://community.pentaho.com/projects/bi_platform. There is also an Enterprise Edition of the platform with additional features and support. You can find more on this at www.pentaho.org. Pentaho Data Integration Most of the Pentaho engines, including the engines mentioned earlier, were created as community projects and later adopted by Pentaho. The PDI engine is not an exception—Pentaho Data Integration is the new denomination for the business intelligence tool born as Kettle. The name Kettle didn’t come from the recursive acronym Kettle Extraction, Transportation, Transformation, and Loading Environment it has now. It came from KDE Extraction, Transportation, Transformation, and Loading Environment, since the tool was planned to be written on top of KDE, a Linux desktop environment, as mentioned in the introduction of the article. In April 2006, the Kettle project was acquired by the Pentaho Corporation and Matt Casters, the Kettle founder, also joined the Pentaho team as a Data Integration Architect. When Pentaho announced the acquisition, James Dixon, Chief Technology Officer said: We reviewed many alternatives for open source data integration, and Kettle clearly had the best architecture, richest functionality, and most mature user interface. The open architecture and superior technology of the Pentaho BI Platform and Kettle allowed us to deliver integration in only a few days, and make that integration available to the community. By joining forces with Pentaho, Kettle benefited from a huge developer community, as well as from a company that would support the future of the project. From that moment, the tool has grown with no pause. Every few months a new release is available, bringing to the users improvements in performance, existing functionality, new functionality, ease of use, and great changes in look and feel. The following is a timeline of the major events related to PDI since its acquisition by Pentaho: June 2006: PDI 2.3 is released. Numerous developers had joined the project and there were bug fixes provided by people in various regions of the world. The version included among other changes, enhancements for large-scale environments and multilingual capabilities. February 2007: Almost seven months after the last major revision, PDI 2.4 is released including remote execution and clustering support, enhanced database support, and a single designer for jobs and transformations, the two main kind of elements you design in Kettle. May 2007: PDI 2.5 is released including many new features; the most relevant being the advanced error handling. November 2007: PDI 3.0 emerges totally redesigned. Its major library changed to gain massive performance. The look and feel had also changed completely. October 2008: PDI 3.1 arrives, bringing a tool which was easier to use, and with a lot of new functionality as well. April 2009: PDI 3.2 is released with a really large amount of changes for a minor version: new functionality, visualization and performance improvements, and a huge amount of bug fixes. The main change in this version was the incorporation of dynamic clustering. June 2010: PDI 4.0 was released, delivering mostly improvements with regard to enterprise features, for example, version control. In the community version, the focus was on several visual improvements such as the mouseover assistance that you will experiment with soon. November 2010: PDI 4.1 is released with many bug fixes. August 2011: PDI 4.2 comes to light not only with a large amount of bug fixes, but also with a lot of improvements and new features. In particular, several of them were related to the work with repositories. April 2012: PDI 4.3 is released also with a lot of fixes, and a bunch of improvements and new features. November 2012: PDI 4.4 is released. This version incorporates a lot of enhancements and new features. In this version there is a special emphasis on Big Data—the ability of reading, searching, and in general transforming large and complex collections of datasets. 2013: PDI 5.0 will be released, delivering interesting low-level features such as step load balancing, job transactions, and restartability. Using PDI in real-world scenarios Paying attention to its name, Pentaho Data Integration, you could think of PDI as a tool to integrate data. In fact, PDI not only serves as a data integrator or an ETL tool. PDI is such a powerful tool, that it is common to see it used for these and for many other purposes. Here you have some examples. Loading data warehouses or datamarts The loading of a data warehouse or a datamart involves many steps, and there are many variants depending on business area, or business rules. But in every case, no exception, the process involves the following steps: Extracting information from one or different databases, text files, XML files and other sources. The extract process may include the task of validating and discarding data that doesn’t match expected patterns or rules. Transforming the obtained data to meet the business and technical needs required on the target. Transformation implies tasks as converting data types, doing some calculations, filtering irrelevant data, and summarizing. Loading the transformed data into the target database. Depending on the requirements, the loading may overwrite the existing information, or may add new information each time it is executed. Kettle comes ready to do every stage of this loading process. The following screenshot shows a simple ETL designed with Kettle: Integrating data Imagine two similar companies that need to merge their databases in order to have a unified view of the data, or a single company that has to combine information from a main ERP (Enterprise Resource Planning) application and a CRM (Customer Relationship Management) application, though they’re not connected. These are just two of hundreds of examples where data integration is needed. The integration is not just a matter of gathering and mixing data. Some conversions, validation, and transport of data have to be done. Kettle is meant to do all of those tasks. Data cleansing It’s important and even critical that data be correct and accurate for the efficiency of business, to generate trust conclusions in data mining or statistical studies, to succeed when integrating data. Data cleansing is about ensuring that the data is correct and precise. This can be achieved by verifying if the data meets certain rules, discarding or correcting those which don’t follow the expected pattern, setting default values for missing data, eliminating information that is duplicated, normalizing data to conform minimum and maximum values, and so on. These are tasks that Kettle makes possible thanks to its vast set of transformation and validation capabilities. Migrating information Think of a company, any size, which uses a commercial ERP application. One day the owners realize that the licenses are consuming an important share of its budget. So they decide to migrate to an open source ERP. The company will no longer have to pay licenses, but if they want to change, they will have to migrate the information. Obviously, it is not an option to start from scratch, nor type the information by hand. Kettle makes the migration possible thanks to its ability to interact with most kind of sources and destinations such as plain files, commercial and free databases, and spreadsheets, among others. Exporting data Data may need to be exported for numerous reasons: To create detailed business reports To allow communication between different departments within the same company To deliver data from your legacy systems to obey government regulations, and so on Kettle has the power to take raw data from the source and generate these kind of ad-hoc reports. Integrating PDI along with other Pentaho tools The previous examples show typical uses of PDI as a standalone application. However, Kettle may be used embedded as part of a process or a dataflow. Some examples are pre-processing data for an online report, sending mails in a scheduled fashion, generating spreadsheet reports, feeding a dashboard with data coming from web services, and so on. The use of PDI integrated with other tools is beyond the scope of this article. If you are interested, you can find more information on this subject in the Pentaho Data Integration 4 Cookbook by Packt Publishing at http://www.packtpub.com/pentaho-data-integration-4-cookbook/book. Installing PDI In order to work with PDI, you need to install the software. It’s a simple task, so let’s do it now. Time for action – installing PDI These are the instructions to install PDI, for whatever operating system you may be using. The only prerequisite to install the tool is to have JRE 6.0 installed. If you don’t have it, please download it from www.javasoft.com and install it before proceeding. Once you have checked the prerequisite, follow these steps: Go to the download page at http://sourceforge.net/projects/pentaho/files/Data Integration. Choose the newest stable release. At this time, it is 4.4.0, as shown in the following screenshot: Download the file that matches your platform. The preceding screenshot should help you. Unzip the downloaded file in a folder of your choice, that is, c:/util/kettle or /home/pdi_user/kettle. If your system is Windows, you are done. Under Unix-like environments, you have to make the scripts executable. Assuming that you chose /home/pdi_user/kettle as the installation folder, execute: cd /home/pdi_user/kettle chmod +x *.sh In Mac OS you have to give execute permissions to the JavaApplicationStub file. Look for this file; it is located in Data Integration 32-bit.appContentsMacOS, or Data Integration 64-bit.appContentsMacOS depending on your system. What just happened? You have installed the tool in just a few minutes. Now, you have all you need to start working. Launching the PDI graphical designer – Spoon Now that you’ve installed PDI, you must be eager to do some stuff with data. That will be possible only inside a graphical environment. PDI has a desktop designer tool named Spoon. Let’s launch Spoon and see what it looks like. Time for action – starting and customizing Spoon In this section, you are going to launch the PDI graphical designer, and get familiarized with its main features. Start Spoon. If your system is Windows, run Spoon.bat You can just double-click on the Spoon.bat icon, or Spoon if your Windows system doesn’t show extensions for known file types. Alternatively, open a command window—by selecting Run in the Windows start menu, and executing cmd, and run Spoon.bat in the terminal. In other platforms such as Unix, Linux, and so on, open a terminal window and type spoon.sh If you didn’t make spoon.sh executable, you may type sh spoon.sh Alternatively, if you work on Mac OS, you can execute the JavaApplicationStub file, or click on the Data Integration 32-bit.app, or Data Integration 64-bit.app icon As soon as Spoon starts, a dialog window appears asking for the repository connection data. Click on the Cancel button. A small window labeled Spoon tips... appears. You may want to navigate through various tips before starting. Eventually, close the window and proceed. Finally, the main window shows up. A Welcome! window appears with some useful links for you to see. Close the window. You can open it later from the main menu. Click on Options... from the menu Tools. A window appears where you can change various general and visual characteristics. Uncheck the highlighted checkboxes, as shown in the following screenshot: Select the tab window Look & Feel. Change the Grid size and Preferred Language settings as shown in the following screenshot: Click on the OK button. Restart Spoon in order to apply the changes. You should not see the repository dialog, or the Welcome! window. You should see the following screenshot full of French words instead: What just happened? You ran for the first time Spoon, the graphical designer of PDI. Then you applied some custom configuration. In the Option… tab, you chose not to show the repository dialog or the Welcome! window at startup. From the Look & Feel configuration window, you changed the size of the dotted grid that appears in the canvas area while you are working. You also changed the preferred language. These changes were applied as you restarted the tool, not before. The second time you launched the tool, the repository dialog didn’t show up. When the main window appeared, all of the visible texts were shown in French which was the selected language, and instead of the Welcome! window, there was a blank screen. You didn’t see the effect of the change in the Grid option. You will see it only after creating or opening a transformation or job, which will occur very soon! Spoon Spoon, the tool you’re exploring in this section, is the PDI’s desktop design tool. With Spoon, you design, preview, and test all your work, that is, Transformations and Jobs. When you see PDI screenshots, what you are really seeing are Spoon screenshots. Setting preferences in the Options window In the earlier section, you changed some preferences in the Options window. There are several look and feel characteristics you can modify beyond those you changed. Feel free to experiment with these settings. Remember to restart Spoon in order to see the changes applied. In particular, please take note of the following suggestion about the configuration of the preferred language. If you choose a preferred language other than English, you should select a different language as an alternative. If you do so, every name or description not translated to your preferred language, will be shown in the alternative language. One of the settings that you changed was the appearance of the Welcome! window at startup. The Welcome! window has many useful links, which are all related with the tool: wiki pages, news, forum access, and more. It’s worth exploring them. You don’t have to change the settings again to see the Welcome! window. You can open it by navigating to Help | Welcome Screen. Storing transformations and jobs in a repository The first time you launched Spoon, you chose not to work with repositories. After that, you configured Spoon to stop asking you for the Repository option. You must be curious about what the repository is and why we decided not to use it. Let’s explain it. As we said, the results of working with PDI are transformations and jobs. In order to save the transformations and jobs, PDI offers two main methods: Database repository: When you use the database repository method, you save jobs and transformations in a relational database specially designed for this purpose. Files: The files method consists of saving jobs and transformations as regular XML files in the filesystem, with extension KJB and KTR respectively. It’s not allowed to mix the two methods in the same project. That is, it makes no sense to mix jobs and transformations in a database repository with jobs and transformations stored in files. Therefore, you must choose the method when you start the tool. By clicking on Cancel in the repository window, you are implicitly saying that you will work with the files method. Why did we choose not to work with repositories? Or, in other words, to work with the files method? Mainly for two reasons: Working with files is more natural and practical for most users. Working with a database repository requires minimal database knowledge, and that you have access to a database engine from your computer. Although it would be an advantage for you to have both preconditions, maybe you haven’t got both of them. There is a third method called File repository, that is a mix of the two above—it’s a repository of jobs and transformations stored in the filesystem. Between the File repository and the files method, the latest is the most broadly used. Therefore, throughout this article we will use the files method. Creating your first transformation Until now, you’ve seen the very basic elements of Spoon. You must be waiting to do some interesting task beyond looking around. It’s time to create your first transformation.

0
0
6774

IBM Tuesday

Oracle E-Business Suite: Creating Bank Accounts and Cash Forecasts

Visualizations Using CCC

REST APIs for social network data using py2neo

Scraping the Data

Customizing Layout with Themes in PHP-Nuke

Working with Spark’s graph processing library, GraphFrames

Getting Started with Meteor

Personalizing Vim

Ridge Regression

Trending Topics

Ext JS 4: Working with the Grid Component

Assessment Planning

Microsoft Dynamics GP: Data Management

Introduction to the Editing Operators in Blender

Getting Started with Pentaho Data Integration

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access