How-To Tutorials

This article by Javier Collado expands the set of web scraping techniques shown in his previous article by looking closely into a more complex problem that cannot be solved with the tools that were explained there. For those who missed out on that article, here's the link. Web Scraping with Python This article will show how to extract the desired information using the same three steps when the web page is not written directly using HTML, but is auto-generated using JavaScript to update the DOM tree. As you may remember from that article, web scraping is the ability to extract information automatically from a set of web pages that were designed only to display information nicely to humans; but that might not be suitable when a machine needs to retrieve that information. The three basic steps that were recommended to be followed when performing a scraping task were the following: Explore the website to find out where the desired information is located in the HTML DOM tree Download as many web pages as needed Parse downloaded web pages and extract the information from the places found in the exploration step What should be taken into account when the content is not directly coded in the HTML DOM tree? The main difference, as you probably have already noted, is that using the downloading methods that were suggested in the previous article (urllib2 or mechanize) just don't work. This is because they generate an HTTP request to get the web page and deliver the received HTML directly to the scraping script. However, the pieces of information that are auto-generated by the JavaScript code are not yet in the HTML file because the code is not executed in any virtual machine as it happens when the page is displayed in a web browser. Hence, instead of relying on a library that generates HTTP requests, we need a library that behaves as a real web browser, or even better, a library that interacts with a real web browser. So that we are sure that we obtain the same data as we see when manually opening a page in a web browser. Please remember that the aim of web scraping is actually parsing the data that a human user sees, so interacting with a real web browser would be a really nice feature. Is there any tool out there to perform that? Fortunately, the answer is yes. In particular, there are a couple of tools used for web testing automation that can be used to solve the JavaScript execution problem: Selenium and Windmill . For the code samples in the sections below, Windmill is used. Any choice would be fine as both of them are well documented and stable tools ready to be used for production. Let's now follow the same three steps that were suggested in the previous article to solve the scraping of the contents of a web page that is partly generated using JavaScript code. Explore Imagine that you are a fan of NASA Image of the day gallery. You want to get a list of the names of all the images in the gallery together with the link to the whole resolution picture just in case you decide to download it later to use as a desktop wallpaper. The first thing to do is to locate the data that has to be extracted on the desired web page. In the case of the Image of the day gallery (see screenshot below), there are three elements that are important to note: Title of the image that is being currently displayed Link to the image full resolution file Next link to make it possible navigate through all the images To find out the location of each piece of interesting information, as it was already suggested in the previous article, it's better to use a tool such as Firebug whose inspect functionality can be really useful. The following picture, for example, shows the location of the image title inside an h3 tag: The other two fields can be located as easily as the title, so no further explanation will be given here. Please refer to the previous article for further information. Download As explained in the introduction, to download the content of the web page, we will use Windmill as it allows the JavaScript code to execute in the web browser before getting the page content. Because Windmill is mostly a testing library, instead of writing a script that calls the Windmill API, I will write a test case for Windmill to navigate through all the image web pages. The code for the test should be as follows: 1 def test_scrape_iotd_gallery(): 2 """ 3 Scrape NASA Image of the Day Gallery 4 """ 5 # Extra data massage for BeautifulSoup 6 my_massage = get_massage() 7 8 # Open main gallery page 9 client = WindmillTestClient(__name__) 10 client.open(url='http://www.nasa.gov/multimedia/imagegallery/iotd.html') 11 12 # Page isn't completely loaded until image gallery data 13 # has been updated by javascript code 14 client.waits.forElement(xpath=u"//div[@id='gallery_image_area']/img", 15 timeout=30000) 16 17 # Scrape all images information 18 images_info = {} 19 while True: 20 image_info = get_image_info(client, my_massage) 21 22 # Break if image has been already scrapped 23 # (that means that all images have been parsed 24 # since they are ordered in a circular ring) 25 if image_info['link'] in images_info: 26 break 27 28 images_info[image_info['link']] = image_info 29 30 # Click to get the information for the next image 31 client.click(xpath=u"//div[@class='btn_image_next']") 32 33 # Print results to stdout ordered by image name 34 for image_info in sorted(images_info.values(), 35 key=lambda image_info: image_info['name']): 36 print ("Name: %(name)sn" 37 "Link: %(link)sn" % image_info) As it can be seen, the usage of Windmill is similar to other libraries such as mechanize. For example, first of all a client object has to be created to interact with the browser, (line 9) and later, the main web page, that is going to be used to navigate through all the information, has to be opened (line 10). Nevertheless, it also includes some facilities that take into account JavaScript code as shown at line 14. In this line, the waits.forElement method has been used to look for DOM element that is filled by the JavaScript code so when that element, in this case the big image in the image gallery, is displayed, the rest of the script can proceed. It is important to note here that the web page processing doesn't start when the page is downloaded (this happens after line 10), but when there's some evidence that JavaScript code has finished the DOM tree manipulation. For navigating through all the pages that contain the information needed, this is just a matter of pressing over the next arrow (line 30). As the images are ordered in a circular buffer, the point when it is decided to stop is when the same image link has been parsed twice (line 25). To execute the script, instead of launching it as we would normally do for a python script, we should call it through the Windmill script to properly initialize the environment: $ windmill firefox test=nasa_iotd.py As it can be seen in the following screenshot, Windmill takes care of opening a browser (Firefox in this case) window and a controller window in which it's possible to see the commands that the script is executing (several clicks on next in the example): The controller window is really interesting because not only does it display the progress of the test cases, but also allows to enter/record actions interactively, which is a nice feature when trying things out. In particular, the recording may be used under some situations to replace Firebug in the exploration step. This is because the captured actions may be stored in a script without spending much time in xpath expressions. For more information about how to use Windmill and the complete API, please refer to the Windmill documentation.

Introduction SOA enablement wraps key application interfaces in services, and integrates it into the SOA. This largely leaves the existing application logic intact, minimizing changes and adding risk only to those components that needed restructuring work to become SOA-ready. While the interfaces are modernized, without subjecting the core application components to a lot of change, the high costs and the various legacy risks associated with the mainframe platform remain. In addition, the performance and scalability of the new interfaces needs to be well-specified and tested, and the additional load they place on the system should be included in any planned capacity upgrades, potentially increasing the overall costs. Reducing or eliminating the legacy mainframe costs and risks via re-host based modernization also helps customers to fund SOA enablement, and the re-architecture phases of legacy modernization, and lay the groundwork for these steps. SOA-enabling a re-hosted application is a much easier process on an open-systems-based, SOA-ready software stack, and a more efficient one as well in terms of system resource utilization and cost. Re-architecting selected components of a re-hosted application based on specific business needs is a lower risk approach than re-architecting the entire applications en masse, and the risk can be further reduced by ensuring that target re-hosting stack provides rugged and transparent integration between re-hosted services and new components. Keeping It Real: Selective re-architecture is all about maximizing ROI by focusing re-architecture investment in the areas with the best pay-off. Undertaking a change from one language or development paradigm to another shouldn't be undertaken lightly—the investment and risks need to be well understood and justified. It is the right investment for components that require frequent maintenance changes but are difficult to maintain, because of poor /structure and layered changes. The payback on re-architecture investment will come from reducing the cost of future maintenance. Similarly, components that need significant functional changes to meet new business requirements can benefit from substantial productivity increase after re-architecture to a more modern development framework with richer tools to support future changes. The payback comes from greater business agility and time-to-market improvements. On the other hand, well-structured and maintainable COBOL components that do not need extensive changes to meet business needs will have very little return to show for the significant re-architecture investment. Leaving them in COBOL on a modern, extensible platform saves significant re-architecture costs that can be invested elsewhere, reduces risk, and shortens payback time. These considerations can help to optimize ROI for medium to large modernization projects where components measure in hundreds or thousands and contain millions or tens of millions lines of code. Re-Hosting Based Modernization For many organizations, mainframe modernization has become a matter of 'how', and not 'if'. Numerous enterprises and public sector organizations choose re-hosting as the first tangible step in their legacy modernization program precisely because it delivers the best ROI in the fastest possible manner, and accelerates the move to SOA enablement and selective re-architecture. Oracle together with our services partners provides a comprehensive re-hosting-based modernization solution that many customers have leveraged for a successful migration of selected applications or complete mainframe environments ranging from a few hundred MIPS to well over 10,000 MIPS. Two key pillars support successful re-hosting projects: Optimal target environment that lowers the Total Cost of Ownership (TCO) by 50–80 percent and maintains mainframe-class Quality of Service (QoS) using open, extensible, SOA-ready, future-proof architecture Predictable, efficient projects delivered by our SI partners with proven methodologies and automated tools Optimal target environment provided by Oracle is powered by proven open systems software stack leveraging Oracle Database and Oracle Tuxedo for a rock-solid, mainframe-class transaction processing (TP) infrastructure closely matching mainframe requirements for online applications. Mainframe-compatible Transaction Processing: Support for IBM CICS or IMS TM applications in native COBOL or C/C++ language containers with mainframe-compatible TP features. RASP: Mainframe-class performance, reliability, and scalability provided by Oracle Real Application Clusters (RAC) and Tuxedo multi-node and multi-domain clustering for load-balancing and high availability despite failure of individual nodes or network links. Workload and System Management: End-to-end transaction and service monitoring to support 24X7 operations management provided by Oracle's Enterprise Manager Grid Control and Tuxedo System and Application Monitor. SOA Enablement and Integration: Extensibility with Web services using Oracle Services Architecture Leveraging Tuxedo (SALT), J2EE integration (using WebLogic-Tuxedo Connector (WTC), Enterprise Service Bus (ESB), Portal, and BPM technologies to enable easy integration of re-hosted applications into modern Service-Oriented Architectures (SOAs). Scalable Platforms and Commodity Hardware: Scalable, Linux/UNIX-based open systems from HP, Dell, Sun, and IBM, providing: Performance on a par with mainframe systems for most workloads at significantly reduced TCO Reliability and workload management similar to mainframe installations, including physical and logical partitioning Robust clustering technologies for high availability and fail-over capabilities within a data center or across the world The diagram below shows conceptual mapping of mainframe environment to compatible open systems infrastructure: Predictable, efficient projects delivered by leading SIs and key modernization specialists use risk-mitigation methodologies, and automated tools honed over numerous projects to address a complete range of Online, Batch, and Data architectures, and the various technologies used in them. These project methodologies and automated tools that support them encompass all phases of a migration project: Preliminary Assessment Study Application Asset Discovery and Analysis Application and Data Conversion (pilot or entire application portfolio) System and Application Integration Test Engineering Regression and Performance Testing Education and Training Operations Migration Switch-Over Combining a proven target architecture stack that is well-matched to the needs of mainframe applications with mature methodologies supported by automated tools has led to a large and growing number of successful re-hosting projects. There is a rising interest to leverage the re-hosting approach to mainframe application modernization, as a way to get off a mainframe fast, and with minimal risk, in a more predictable manner for large, business-critical applications evolved over a long term and multiple development teams. Re-hosting based modernization approach preserves an organizations long term investment in critical business logic and data without risking business operations or sacrificing the QoS, while enabling customers to: Reduce or eliminate mainframe maintenance costs, and/or defer upgrade costs, saving customers 50–80 percent of their annual maintenance and operations budget Increase productivity and flexibility in IT development and operations, protecting long-term investment through application modernization Speed up and simplify application integration via SOA, without losing transactional integrity and the high performance expected by the users The rest of this article explores the critical success factors and proven transformation architecture for re-hosting legacy applications and data, describes SOA integration options and considerations when SOA-enabling re-hosted applications, highlights key risk mitigation methodologies, and provides a foundation for the financial analysis and ROI model derived from over a hundred, mainframe re-hosting projects. Critical Success Factors in Mainframe Re-Hosting Companies considering a re-hosting-based modernization strategy that involves migrating some applications off the mainframe have to address a range of concerns, which can be summarized by the following questions: How to preserve the business logic of these applications and their valuable data? How to ensure that migrated applications continue to meet performance requirements? How to maintain scalability, reliability, transactional integrity, and other QoS attributes in an open system environment? How to migrate in phases, maintaining robust integration links between migrated and mainframe applications? How to achieve predictable, cost-effective results and ensure a low-risk project? Meeting these challenges requires a versatile and powerful application infrastructure—one that natively supports key mainframe languages and services, enables automated adaptation of application code, and delivers proven, mainframe-like QoS on open system platforms. For re-hosting to enable broader aspects of the modernization strategy, this infrastructure must also provide native Web services and ESB capabilities to rapidly integrate re-hosted applications as first-class services in an SOA. Equally important is a proven, risk-mitigation methodology, automated tools, and project services specifically honed to address automated conversion and adaptation of application code and data, supported by cross-platform test engineering and execution methodology, strong system and application integration expertise, and deep experience with operations migration and switch-over. Preserving Application Logic and Data The re-hosting approach depends on a mainframe-compatible transaction processing and application services platform supporting common mainframe languages such as COBOL and C, which preserves the original business logic and data for the majority of mainframe applications and avoids the risks and uncertainties of a re-write. A complete re-hosting solution provides native support for TP and Batch programs, leveraging an application server-based platform that provides container-based support for COBOL and C/C++ application services, and TP APIs similar to IBM CICS, IMS TM, or other mainframe TP monitors. Online Transaction Processing Environment Oracle Tuxedo is the most popular TP platform for open systems, as well as leading re-hosting platform that can run most of mainframe COBOL and C applications unchanged in container-based framework that combines common application server features, including health monitoring, fail-over, service virtualization, and dynamic load balancing critical to large-scale OLTP applications together with standard TP features, including transaction management and reliable coordination of distributed transactions (a.k.a. Two-Phase Commit or XA standard). It provides the highest possible performance and scalability, and has been recently benchmarked against a mainframe at over 100,000 transactions per second, with sub-second response time. Oracle Tuxedo supports common mainframe programming languages, that is, COBOL and C, and provides comprehensive TP features compatible with CICS and IMS TM, which makes it a preferred application platform choice for re-hosting CICS or IMS TM applications with minimal changes and risks. In the Tuxedo environment, COBOL or C business logic remains unchanged. The only adaptation required is automated mapping of CICS APIs (CICS EXEC calls) to equivalent Tuxedo API functions. This mapping typically leverages a pre-processor and a mapping library implemented on Tuxedo platform, and using a full range of Tuxedo APIs. The automated nature of pre-processing and comprehensive coverage provided by the library ensures that most CICS COBOL or C programs are easily transformed into Tuxedo services. Unlike other solutions that embed this transformation in their compiler coupled with a proprietary emulation run-time, Tuxedo-based solution provides this mapping as a compiler-independent source module, which can be easily extended as needed. The resultant code uses Tuxedo API at native speed, allowing it to reach tens of thousands of transactions per second, while taking advantage of all Tuxedo facilities. In a re-hosted application CICS transactions become Tuxedo services, registered for processing by Tuxedo server processes. These services can be deployed in a single machine or across multiple machines in a Tuxedo domain (SYSPLEX-like cluster.). The services are called by front-end Java, .Net, or Tuxedo/WS clients, or UI components (tn3270 or web-based converted 3270/BMS screens), or by other services in case of transaction linking. Deferred transactions are handled by Tuxedo's/Q component, which provides in-memory and persistent queuing services. The diagram below shows Oracle Tuxedo and its surrounding ecosystem of SOA, J2EE, ESB, CORBA, MQ, and Mainframe integration components:   User Interface Migration The UI elements in these programs are typically defined using CICS Basic Mapping Support (BMS) for 3270 "green screen" terminals. While it is possible to preserve these using tn3270 emulation, many customers in re-hosting projects choose to take advantage of automated conversion of BMS macros into JSP/HTML for Web UI. Supported by a specialized Javascript library, these Web screens mimic the appearance and the behavior of "green screens" in a web browser, including tab-based navigation and PF keys. These UI components can connect to re-hosted CICS transactions running as Tuxedo services using Oracle Jolt (Java client interface for Tuxedo), Weblogic-Tuxedo Connector (WTC), or Tuxedo's Web services gateway provided by Oracle Services Architecture Leveraging Tuxedo (SALT) product. The diagram on the next page depicts a target re-hosting architecture for a typical mainframe OLTP application. The architecture uses Tuxedo services to run re-hosted CICS programs and a web application server to run re-hosted BMS UI. The servlets or JSPs containing the HTML that defines the screens, connect with Tuxedo services via Oracle Jolt, WTC, or SALT. Customers using mainframe 4GLs or languages such as PL/I or Assembler frequently choose to convert these applications to COBOL or C/C++. The adaptation of CICS or IMS TM API calls is automated through a mapping layer, which minimizes overall changes for the development team and allows them to maintain the familiar applications. For more significant extensions and new capabilities, customers incrementally leverage Tuxedo's own APIs and facilities, or leverage a tightly-linked J2EE environment provided by the WebLogic Server, and even transparently make Web services calls. The optimal extensibility options depend on application needs, availability of Java or C/COBOL skills, and other factors.   Feature or Action CICS Verb Tuxedo API Communications Area DFHCOMMAREA Typed Buffer Transaction Request LINK tpcall Transaction Return RETURN tpreturn Transfer Control XCTL tpforward Allocate Storage GETMAIN tpalloc Queues READQ / WRITEQ TD,TS /Q tpenqueue / tpdequeue Begin new transaction START TRANID /Q and TMQFORWARD Abort transaction ISSUE ABEND tpreturn TPFAIL Commit or Rollback SYNCPOINT / SYNCPOINT ROLLBACK tpcommit / tpabort     Keeping it Real:For those familiar with CICS, this is a very short example of the CICS verbs. CICS has many functions, most of which either map natively to a similar Tuxedo API or are provided by migration specialists based on their extensive experience with such migrations. In summary, Tuxedo provides a popular platform for deploying, executing, and managing COBOL and C re-hosted transactional applications requiring any of the following OLTP and infrastructure services: Native, compiler-independent support for COBOL, C, or C++ Rich set of infrastructure services for managing and scaling diverse workloads Feature-set compatibility and inter-operability with IBM CICS and IMS/TM Two-Phase Commit (2PC) for managing transactions across multiple application domains and XA-compliant resource managers (databases, message queues) Guaranteed inter-application messaging and transactional queuing Transactional data access (using XA-compliant resource managers) with ACID qualities Services virtualization and dynamic load balancing Centralized management of multiple nodes in a domain, and across multiple domains Communications gateways for multiple traditional and modern communication protocols SOA Enablement through native Web services and ESB integration Workload Monitoring and Management An important aspect of the mainframe environment is workload monitoring and management, which provides information for effective performance analysis and capabilities that enable mainframe systems to achieve better throughput and responsiveness. Oracle's Tuxedo System and Application Monitor (TSAM) provides similar capabilities too. Define monitoring policies and patterns based on application requests, services, system servers such as gateways, bridges, and XA-defined stages of a distributed transaction Define SLA thresholds that can trigger a variety of events within Tuxedo event services including notifications, and instantiation of additional servers Monitor transactions on an end-to-end basis from a client call through all services across all domains involved in a client request Collect service statistics for all infrastructure components such as servers and gateways Detail time spent on IPC queues, waiting on network links, and time spent on subordinate services TSAM provides a built-in, central, web-based management and monitoring console, and an open framework for integration with third-party performance management tools. Batch Jobs Mainframe batch jobs are a response to a human 24-hour clock on which many businesses run. It includes beginning-of-period or end-of-period (day, week, month, quarter) processing for batched updates, reconciliation, reporting, statement generation, and similar applications. In some industries, external events tied to a fixed schedule such as intra-day, opening or closing trade in a stock exchange, drive specific processing needs. Batch applications are an equally important asset, and often need to be preserved and migrated as well. The batch environment uses Job Control Language (JCL) jobs managed and monitored by JES2 or JES3 (Job Entry System), which invoke one or more programs, access and manipulate large datasets and databases using sort and other specialized utilities, and often run under the control of a job scheduler such as CA-7/CA-11. JCL defines a series of job steps—a sequence of programs and utilities, specifies input and output files, and provides exception handling. Automated parsing and translation of JCL jobs to UNIX scripts such as Korn shell (ksh) or Perl, enables the overall structure of the job to remain the same, including job steps, classes, and exception handling. Standard shell processing is supplemented with required utilities such as SyncSort, and support for Generation Data Group (GDG) files. REXX/CLIST/PROC scripting environments on the mainframe are similarly converted to ksh or other scripting languages. Integration with Oracle Scheduler, or other job schedulers running in UNIX/Linux or Windows provides a rich set of calendar and event-based scheduling capabilities as well as dependency management similar to mainframe schedulers. In some cases, reporting done via batch jobs can be replaced using standard reporting packages such as Oracle BI Publisher. The diagram below shows a typical target re-hosting architecture for batch. It includes a scheduler to control and trigger batch jobs, scripting framework to support individual job scripts, and an application server execution framework for the batch COBOL or C programs. Unlike other solutions that run these programs directly as OS processes without the benefit of application server middleware, Oracle recommends using container-based middleware to provide higher reliability, availability, and monitoring to the batch programs. The target batch programs invoked by the scripts can also run directly as OS processes, but if mainframe-class management and monitoring similar to JES2 or JES3 environment is a requirement, these programs can run as services under Tuxedo, benefiting from the health monitoring, fail-over, load balancing, and other application server-like features it provides. Files and Databases When moving platforms (mainframe to open systems), the application and data have to be moved together. Data schemas and data stores need to be moved in a re-hosted mainframe modernization project just as with a re-architecture. The approach taken depends on the source data store. DB2 is the most straightforward, since DB2 and Oracle are both relational databases. In addition to migrating the data, customers sometimes choose to perform data cleansing, field extensions, merge columns, or other data maintenance practices leveraging the automated tooling that synchronizes all data changes with changes to the application's data access code. Mainframe DB2 DB2 is a predominant relational database on IBM mainframes. When migrating to Oracle Database, the migration approach is highly automated, and resolves all discrepancies between the two RDBMS in terms of field formats as well as error codes returned to applications, so as to maintain application behavior unchanged, including stored procedures if any. IMS IMS/DB (also known as DL/1) is a popular hierarchical database for older applications. Creating appropriate relational data schema for this data requires an understanding of the application access patterns so as to optimize the schema for best performance based on the most frequent access paths. To minimize code impact, a translation layer can be used at run-time to support IMS DB style data access from the application, and map it to appropriate SQL calls. This allows the applications to interface with the segments, now translated as DB2 UDB or ORACLE tables, without impacting application code and maintenance. VSAM VSAM files are used for keyed-sequential data access, and can be readily migrated to ISAM files or to Oracle Database tables wherever transactional integrity is required (XA features). Some customers also choose to migrate VSAM files to Oracle Database to provide accessibility from other distributed applications, or to simplify the re-engineering required to extend certain data fields or merge multiple data sources. Meeting Performance and Other QoS Requirements The mainframe's performance, reliability, scalability, manageability, and other QoS attributes have earned it pre-eminence for business-critical applications. How well do re-hosting solutions measure up against these characteristics? Earlier solutions based on IBM CICS emulators derived from development tools often did not measure up to the demands of mainframe workloads since they were never intended for true production environment and have not been exposed to large-scale applications. As a result, they have only been used for re-hosting small systems under 300 MIPS and not requiring any clustering or distributed workload handling. Oracle Tuxedo was built to scale ground up, to support high performance telecommunications operations. It has the distinction of being the only non-mainframe TP solution recognized for its mainframe-like performance, reliability, and QoS characteristics. Most large enterprise customers requiring such capabilities in distributed systems have traditionally relied on Tuxedo. Consistently rated by IDC and Gartner as the market leader, and predominant in non-mainframe OLTP applications, it has also become the preferred COBOL/C application platform and transaction engine for re-hosted mainframe applications requiring high performance and/or mission-critical availability and reliability. Reasons for the broad recognition of Tuxedo as the only mainframe-class application platform and transaction engine for distributed systems are based on mainframe-class performance, scalability, reliability, availability, and other QoS attributes proven in multiple customer deployments. The following table highlights some of these capabilities:   Reliability Availability Guaranteed messaging and transactional integrity Hardened code from 25 years of use in the world's largest transaction applications Transaction integrity across systems and domains through a two phase commit (XA) for all resources such as databases, queues, and so on. Proven in mainframe-to-mainframe transactions and messaging No single point of failure, 99.999% uptime with N+1/N+2 clusters Application services upgradeable in operation Self-monitoring, automated fail-over, datadriven routing for super high availability Centralized monitoring and management with clustered domains; automated, lights-out operations     Workload Management   Performance and Scalability   Resource management and prioritization across Tuxedo services Dynamic load balancing across domains based on load conditions Data-driven routing enables horizontally distributed database grids and differentiated QoS End-to-end monitoring of Tuxedo system and application services enables SLA enforcement Virtualization support enables spawning of Tuxedo servers on demand Parallel processing to maximize resource utilization with low latency code paths that provide sub-second response at any load Horizontal and vertical scaling of system resources yields linear performance increases Request multiplexing (synchronous and asynchronous) maximizes CPU utilization Proven in credit card authorizations at over 13.5K tps, and in telco billing at over 56K tps. Middleware of choice in HP, Fujitsu, Sun, IBM, and NEC TPC-C benchmarks    

We will look into: Creating our first JPA entity Interacting with JPA entities with entity manager Generating forms in JSF pages from JPA entities Generating JPA entities from an existing database schema JPA named queries and JPQL Entity relationships Generating complete JSF applications from JPA entities Creating Our First JPA Entity JPA entities are Java classes whose fields are persisted to a database by the JPA API. JPA entities are Plain Old Java Objects (POJOs), as such, they don't need to extend any specific parent class or implement any specific interface. A Java class is designated as a JPA entity by decorating it with the @Entity annotation. In order to create and test our first JPA entity, we will be creating a new web application using the JavaServer Faces framework. In this example we will name our application jpaweb. As with all of our examples, we will be using the bundled GlassFish application server. To create a new JPA Entity, we need to right-click on the project and select New | Entity Class. After doing so, NetBeans presents the New Entity Class wizard. At this point, we should specify the values for the Class Name and Package fields (Customer and com.ensode.jpaweb in our example), then click on the Create Persistence Unit... button. The Persistence Unit Name field is used to identify the persistence unit that will be generated by the wizard, it will be defined in a JPA configuration file named persistence.xml that NetBeans will automatically generate from the Create Persistence Unit wizard. The Create Persistence Unit wizard will suggest a name for our persistence unit, in most cases the default can be safely accepted. JPA is a specification for which several implementations exist. NetBeans supports several JPA implementations including Toplink, Hibernate, KODO, and OpenJPA. Since the bundled GlassFish application server includes Toplink as its default JPA implementation, it makes sense to take this default value for the Persistence Provider field when deploying our application to GlassFish. Before we can interact with a database from any Java EE 5 application, a database connection pool and data source need to be created in the application server. A database connection pool contains connection information that allow us to connect to our database, such as the server name, port, and credentials. The advantage of using a connection pool instead of directly opening a JDBC connection to a database is that database connections in a connection pool are never closed, they are simply allocated to applications as they need them. This results in performance improvements, since the operations of opening and closing database connections are expensive in terms of performance. Data sources allow us to obtain a connection from a connection pool by obtaining an instance of javax.sql.DataSource via JNDI, then invoking its getConnection() method to obtain a database connection from a connection pool. When dealing with JPA, we don't need to directly obtain a reference to a data source, it is all done automatically by the JPA API, but we still need to indicate the data source to use in the application's Persistence Unit. NetBeans comes with a few data sources and connection pools pre-configured. We could use one of these pre-configured resources for our application, however, NetBeans also allows creating these resources "on the fly", which is what we will be doing in our example. To create a new data source we need to select the New Data Source... item from the Data Source combo box. A data source needs to interact with a database connection pool. NetBeans comes pre-configured with a few connection pools out of the box, but just like with data sources, it allows us to create a new connection pool "on demand". In order to do this, we need to select the New Database Connection... item from the Database Connection combo box. NetBeans includes JDBC drivers for a few Relational Database Management Systems (RDBMS) such as JavaDB, MySQL, and PostgreSQL "out of the box". JavaDB is bundled with both GlassFish and NetBeans, therefore we picked JavaDB for our example. This way we avoid having to install an external RDBMS. For RDBMS systems that are not supported out of the box, we need to obtain a JDBC driver and let NetBeans know of it's location by selecting New Driver from the Name combo box. We then need to navigate to the location of a JAR file containing the JDBC driver. Consult your RDBMS documentation for details. JavaDB is installed in our workstation, therefore the server name to use is localhost. By default, JavaDB listens to port 1527, therefore that is the port we specify in the URL. We wish to connect to a database called jpaintro, therefore we specify it as the database name. Since the jpaintro database does not exist yet, we pass the attribute create=true to JavaDB, this attribute is used to create the database if it doesn't exist yet. Every JavaDB database contains a schema named APP, since each user by default uses a schema named after his/her own login name. The easiest way to get going is to create a user named "APP" and select a password for this user. Clicking on the Show JDBC URL checkbox reveals the JDBC URL for the connection we are setting up. The New Database Connection wizard warns us of potential security risks when choosing to let NetBeans remember the password for the database connection. Database passwords are scrambled (but not encrypted) and stored in an XML file under the .netbeans/[netbeans version]/config/Databases/Connections directory. If we follow common security practices such as locking our workstation when we walk away from it, the risks of having NetBeans remember database passwords will be minimal. Once we have created our new data source and connection pool, we can continue configuring our persistence unit. It is a good idea to leave the Use Java Transaction APIs checkbox checked. This will instruct our JPA implementation to use the Java Transaction API (JTA) to allow the application server to manage transactions. If we uncheck this box, we will need to manually write code to manage transactions. Most JPA implementations allow us to define a table generation strategy. We can instruct our JPA implementation to create tables for our entities when we deploy our application, to drop the tables then regenerate them when our application is deployed, or not create any tables at all. NetBeans allows us to specify the table generation strategy for our application by clicking the appropriate value in the Table Generation Strategy radio button group. When working with a new application, it is a good idea to select the Drop and Create table generation strategy. This will allow us to add, remove, and rename fields in our JPA entity at will without having to make the same changes in the database schema. When selecting this table generation strategy, tables in the database schema will be dropped and recreated, therefore any data previously persisted will be lost. Once we have created our new data source, database connection and persistence unit, we are ready to create our new JPA entity. We can do so by simply clicking on the Finish button. At this point NetBeans generates the source for our JPA entity. JPA allows the primary field of a JPA entity to map to any column type (VARCHAR, NUMBER). It is best practice to have a numeric surrogate primary key, that is, a primary key that serves only as an identifier and has no business meaning in the application. Selecting the default Primary Key type of long will allow for a wide range of values to be available for the primary keys of our entities. package com.ensode.jpaweb;import java.io.Serializable;import javax.persistence.Entity;import javax.persistence.GeneratedValue;import javax.persistence.GenerationType;import javax.persistence.Id;@Entitypublic class Customer implements Serializable { private static final long serialVersionUID = 1L; private Long id; public void setId(Long id) { this.id = id; } @Id @GeneratedValue(strategy = GenerationType.AUTO) public Long getId() { return id; } //Other generated methods (hashCode(), equals() and //toString() omitted for brevity.} As we can see, a JPA entity is a standard Java object. There is no need to extend any special class or implement any special interface. What differentiates a JPA entity from other Java objects are a few JPA-specific annotations. The @Entity annotation is used to indicate that our class is a JPA entity. Any object we want to persist to a database via JPA must be annotated with this annotation. The @Id annotation is used to indicate what field in our JPA entity is its primary key. The primary key is a unique identifier for our entity. No two entities may have the same value for their primary key field. This annotation can be placed just above the getter method for the primary key class. This is the strategy that the NetBeans wizard follows. It is also correct to specify the annotation right above the field declaration. The @Entity and the @Id annotations are the bare minimum two annotations that a class needs in order to be considered a JPA entity. JPA allows primary keys to be automatically generated. In order to take advantage of this functionality, the @GeneratedValue annotation can be used. As we can see, the NetBeans generated JPA entity uses this annotation. This annotation is used to indicate the strategy to use to generate primary keys. All possible primary key generation strategies are listed in the following table:   Primary Key Generation Strategy   Description   GenerationType.AUTO   Indicates that the persistence provider will automatically select a primary key generation strategy. Used by default if no primary key generation strategy is specified.   GenerationType.IDENTITY   Indicates that an identity column in the database table the JPA entity maps to must be used to generate the primary key value.   GenerationType.SEQUENCE   Indicates that a database sequence should be used to generate the entity's primary key value.   GenerationType.TABLE   Indicates that a database table should be used to generate the entity's primary key value.       In most cases, the GenerationType.AUTO strategy works properly, therefore it is almost always used. For this reason the New Entity Class wizard uses this strategy. When using the sequence or table generation strategies, we might have to indicate the sequence or table used to generate the primary keys. These can be specified by using the @SequenceGenerator and @TableGenerator annotations, respectively. Consult the Java EE 5 JavaDoc at http://java.sun.com/javaee/5/docs/api/ for details. For further knowledge on primary key generation strategies you can refer EJB 3 Developer Guide by Michael Sikora, which is another book by Packt Publishing (http://www.packtpub.com/developer-guide-for-ejb3/book). Adding Persistent Fields to Our Entity At this point, our JPA entity contains a single field, its primary key. Admittedly not very useful, we need to add a few fields to be persisted to the database. package com.ensode.jpaweb;import java.io.Serializable;import javax.persistence.Entity;import javax.persistence.GeneratedValue;import javax.persistence.GenerationType;import javax.persistence.Id;@Entitypublic class Customer implements Serializable { private static final long serialVersionUID = 1L; private Long id; private String firstName; private String lastName; public void setId(Long id) { this.id = id; } @Id @GeneratedValue(strategy = GenerationType.AUTO) public Long getId() { return id; } public String getFirstName() { return firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } public String getLastName() { return lastName; } public void setLastName(String lastName) { this.lastName = lastName; } //Additional methods omitted for brevity} In this modified version of our JPA entity, we added two fields to be persisted to the database; firstName will be used to store the user's first name, lastName will be used to store the user's last name. JPA entities need to follow standard JavaBean coding conventions. This means that they must have a public constructor that takes no arguments (one is automatically generated by the Java compiler if we don't specify any other constuctors), and all fields must be private, and accessed through getter and setter methods. Automatically Generating Getters and Setters In NetBeans, getter and setter methods can be generated automatically. Simply declare new fields as usual then use the "insert code" keyboard shortcut (default is Alt+Insert), then select Getter and Setter from the resulting pop-up window, then click on the check box next to the class name to select all fields, then click on the Generate button. Before we can use JPA persist our entity's fields into our database, we need to write some additional code. Creating a Data Access Object (DAO) It is a good idea to follow the DAO design pattern whenever we write code that interacts with a database. The DAO design pattern keeps all database access functionality in DAO classes. This has the benefit of creating a clear separation of concerns, leaving other layers in our application, such as the user interface logic and the business logic, free of any persistence logic. There is no special procedure in NetBeans to create a DAO. We simply follow the standard procedure to create a new class by selecting File | New, then selecting Java as the category and the Java Class as the file type, then entering a name and a package for the class. In our example, we will name our class CustomerDAO and place it in the com.ensode.jpaweb package. At this point, NetBeans create a very simple class containing only the package and class declarations. To take complete advantage of Java EE features such as dependency injection, we need to make our DAO a JSF managed bean. This can be accomplished by simply opening faces-config.xml, clicking its XML tab, then right-clicking on it and selecting JavaServer Faces | Add Managed Bean. We get the Add Manged Bean dialog as seen here: We need to enter a name, fully qualified name, and scope for our managed bean (which, in our case, is our DAO), then click on the Add button. This action results in our DAO being declared as a managed bean in our application's faces-config.xml configuration file. <managed-bean> <managed-bean-name>CustomerDAO</managed-bean-name> <managed-bean-class> com.ensode.jpaweb.CustomerDAO </managed-bean-class> <managed-bean-scope>session</managed-bean-scope> </managed-bean> We could at this point start writing our JPA code manually, but with NetBeans there is no need to do so, we can simply right-click on our code and select Persistence | Use Entity Manager, and most of the work is automatically done for us. Here is how our code looks like after doing this trivial procedure: package com.ensode.jpaweb;import javax.annotation.Resource;import javax.naming.Context;import javax.persistence.EntityManager;import javax.persistence.PersistenceContext;@PersistenceContext(name = "persistence/LogicalName", unitName = "jpawebPU")public class CustomerDAO { @Resource private javax.transaction.UserTransaction utx; protected void persist(Object object) { try { Context ctx = (Context) new javax.naming.InitialContext(). lookup("java:comp/env"); utx.begin(); EntityManager em = (EntityManager) ctx.lookup("persistence/LogicalName"); em.persist(object); utx.commit(); } catch (Exception e) { java.util.logging.Logger.getLogger( getClass().getName()).log( java.util.logging.Level.SEVERE, "exception caught", e); throw new RuntimeException(e); } }} All highlighted code is automatically generated by NetBeans. The main thing NetBeans does here is add a method that will automatically insert a new row in the database, effectively persisting our entity's properties. As we can see, NetBeans automatically generates all necessary import statements. Additionally, our new class is automatically decorated with the @PersistenceContext annotation. This annotation allows us to declare that our class depends on an EntityManager (we'll discuss EntityManager in more detail shortly). The value of its name attribute is a logical name we can use when doing a JNDI lookup for our EntityManager. NetBeans by default uses persistence/LogicalName as the value for this property. The Java Naming and Directory Interface (JNDI) is an API we can use to obtain resources, such as database connections and JMS queues, from a directory service. The value of the unitName attribute of the @PersistenceContext annotation refers to the name we gave our application's Persistence Unit. NetBeans also creates a new instance variable of type javax.transaction.UserTransaction. This variable is needed since all JPA code must be executed in a transaction. UserTransaction is part of the Java Transaction API (JTA). This API allows us to write code that is transactional in nature. Notice that the UserTransaction instance variable is decorated with the @Resource annotation. This annotation is used for dependency injection. in this case an instance of a class of type javax.transaction.UserTransaction will be instantiated automatically at run-time, without having to do a JNDI lookup or explicitly instantiating the class. Dependency injection is a new feature of Java EE 5 not present in previous versions of J2EE, but that was available and made popular in the Spring framework. With standard J2EE code, it was necessary to write boilerplate JNDI lookup code very frequently in order to obtain resources. To alleviate this situation, Java EE 5 made dependency injection part of the standard. The next thing we see is that NetBeans added a persist method that will persist a JPA entity, automatically inserting a new row containing our entity's fields into the database. As we can see, this method takes an instance of java.lang.Object as its single parameter. The reason for this is that the method can be used to persist any JPA entity (although in our example, we will use it to persist only instances of our Customer entity). The first thing the generated method does is obtain an instance of javax.naming.InitialContext by doing a JNDI lookup on java:comp/env. This JNDI name is the root context for all Java EE 5 components. The next thing the method does is initiate a transaction by invoking uxt.begin(). Notice that since the value of the utx instance variable was injected via dependency injection (by simply decorating its declaration with the @Resource annotation), there is no need to initialize this variable. Next, the method does a JNDI lookup to obtain an instance of javax.persistence.EntityManager. This class contains a number of methods to interact with the database. Notice that the JNDI name used to obtain an EntityManager matches the value of the name attribute of the @PersistenceContext annotation. Once an instance of EntityManager is obtained from the JNDI lookup, we persist our entity's properties by simply invoking the persist() method on it, passing the entity as a parameter to this method. At this point, the data in our JPA entity is inserted into the database. In order for our database insert to take effect, we must commit our transaction, which is done by invoking utx.commit(). It is always a good idea to look for exceptions when dealing with JPA code. The generated method does this, and if an exception is caught, it is logged and a RuntimeException is thrown. Throwing a RuntimeException has the effect of rolling back our transaction automatically, while letting the invoking code know that something went wrong in our method. The UserTransaction class has a rollback() method that we can use to roll back our transaction without having to throw a RunTimeException. At this point we have all the code we need to persist our entity's properties in the database. Now we need to write some additional code for the user interface part of our application. NetBeans can generate a rudimentary JSF page that will help us with this task.

XML Security Threats All the components in web services are described in XML. SOAP and all the WS -Security specifications are XML formats. Hence it just makes sense for expressing security data in XML format. Fortunately, there has been no need to invent new cryptography technologies for XML. The XML security standards have used existing cryptography directly. XML-based data transfer has emerged as the standard for organizations to exchange business data. As with all communications over the public Internet, XML-based transfers have their own set of vulnerabilities to confront. Like any other document exchange, XML document exchange must support the usual security measures which are Confidentiality, Integrity, Authenticity, and Non-Repudiation. The following list illustrates some specific XML security threats: Schema Altering — Manipulation of WS schema to alter the data processed by the application.      XML Parameter Tampering — Injection of malicious scripts or content into XML parameters      Coercive Parsing — Injection of malicious content into the XML      Oversized Payload — Sending oversized files to create an XDoS attack      Recursive Payload — Sending mass amounts of nested data to create an XDoS attack against an XML parser      XML Routing Detours — Redirecting sensitive data within the XML path      External Entity Attack — An attack on an application that parses XML input from suspicious sources using an incorrectly configured XML parser These threats pose potentially serious problems to developers creating applications, components, and systems that depend on XML data. The solution for the above problems is XML Encryption. XML Encryption XML Encryption provides end-to-end security for applications that require secure exchange of structured data. XML itself is the most popular technology for structuring data, and therefore XML-based encryption is the natural way to handle complex requirements for security in data interchange applications. XML Encryption is a process for encrypting and decrypting parts of XML documents. Most of today's encryption schemes use transport-level techniques that encrypt an entire request and response stream between a sender and receiver, offering zero visibility into contents of the interchange to intermediaries. Contentlevel encryption converts document fragments into illegible ciphertext, while other elements remain legible as plaintext. Some features of XML encryption are: The ability to encrypt a complete XML file The ability to encrypt a single element of an XML file The ability to encrypt only the contents of an XML element The ability to encrypt binary data within an XML file Encrypting an XML File Here's a short sample XML file that can serve to demonstrate XML encryption:     <?xml version='1.0'?>    <POInfo >        <Name>FJ</Name>        <Amount>125.00</Amount>        <CreditCardNumber>1234-5678-4564-4321</CreditCardNumber>    <Date>July 6, 2006</Date>    </POInfo> When you encrypt an entire XML file, the process simply replaces the root element (<POInfo> in the sample) with an <EncryptedData> element that contains the encryption details, including the encrypted content. Here is how the encrypted file will look:     <?xml version="1.0" encoding="UTF-8"?>    <xenc:EncryptedData                Type="http://www.w3.org/2001/04/xmlenc#Element">        <xenc:EncryptionMethod            Algorithm="http://www.w3.org/2001/04/xmlenc#aes128-cbc"            />        <ds:KeyInfo >            <xenc:EncryptedKey                >                <xenc:EncryptionMethod                    Algorithm="http://www.w3.org/2001/04/xmlenc#kw-tripledes"                    />                <xenc:CipherData                    >                    <xenc:CipherValue                        >                        MKeT0ZmHFLwnZaSXO+oZSxlSJ5/BqvblqG76B3nOMU0=                    </xenc:CipherValue>                </xenc:CipherData>            </xenc:EncryptedKey>        </ds:KeyInfo>        <xenc:CipherData            >            <xenc:CipherValue                >                    +M/Tamk/62Lut4HqLpU/es9sdhnNTTpasbeszN8GN8EAJZsX0vvClcKEW                    UAgIdbvyJpprQ+jUIiWJKTz1X3L6VAefHqO963pU3bzmGMo                    pHLqS1Eg7iAPFhKV1PJclyswyyepEjyu+bOgqzgGnS1XA0/V                    NP7kLK70rB2Zb0DSbaCi+7HjTNGWF9YKtPIP5bvrs5xw+x                    HnKO++2EuqzK+deD7mCu8w6sG9vmRCrUR99Mx1QDZon9a2962ZD                    FSwoIJKg5I83GzOU+RObBBUme+yTf7UWybEiwtHp5ZgvuaQYJA=            </xenc:CipherValue>        </xenc:CipherData>    </xenc:EncryptedData> Encrypting a Single Element To encrypt a single element of an XML file, you specify the desired child element, rather than the root element of the input file as the element to encrypt. The following snippet shows the results of encrypting only the <CreditCardNumber> element of the sample file.     <?xml version="1.0" encoding="UTF-8"?>    <POInfo >        <Name>John Doe</Name>        <Amount>125.00</Amount>        <xenc:EncryptedData                        Type="http://www.w3.org/2001/04/xmlenc#Element">            <xenc:EncryptionMethod                Algorithm="http://www.w3.org/2001/04/xmlenc#aes128-cbc"                />            <ds:KeyInfo >                <xenc:EncryptedKey                    >                    <xenc:EncryptionMethod                        Algorithm=                            "http://www.w3.org/2001/04/xmlenc#kw-tripledes"                        />                    <xenc:CipherData                        >                        <xenc:CipherValue                            >                                6zhAcEW7KIKrbSjEOkXDrVkmws5zhQQLDO4YYW+RfRY=                        </xenc:CipherValue>                    </xenc:CipherData>                </xenc:EncryptedKey>            </ds:KeyInfo>            <xenc:CipherData                >                <xenc:CipherValue                    >                    JqsRmdSoS+PXqCe80Y8zNiQ49sHTLNaAgHX1Ja7d+u9fv                    TFBrkBMK7C7EHsQTglZ3yT9yCZDuFnjBoQTLULKqOy71Qw                    EPRPObtYLPIJgy1vUdNrw47uDmJ/R5r/B0SH37HN8mfNv                    i50zPt1qPxxRwA==                </xenc:CipherValue>            </xenc:CipherData>        </xenc:EncryptedData>        <Date>July 6, 2005</Date>    </POInfo> Notice that the encryption process replaced the <CreditCardNumber> tag and its contents with an <EncryptedData> tag, while leaving the siblings of the <CreditCardNumber> element unaltered. This type of encryption can be performed using XML Signature and Encryption. The interested reader may look up the implementation at the Apache site (http://xml.apache.org/security/). Best practices for XML encryption, can be summarized as follows: It is good to have standard element tags for representing encrypted elements within the XML documents. This will enable parsers to better understand encrypted elements and data during the validation process.      It is necessary to provide means for encrypting only the desired elements within an XML document instead of encrypting the whole document. This will pave the way for incorporating several confidential data elements that are intended for different recipients within a single XML document.      There should be standard mechanisms for exchanging the secret keys used for encryption and decryption processes.      The standard should allow encryption of different parts of the document with different keys, so that multiple recipients can decrypt only those portions that are intended for them.      The standards should be adaptable to both ASCII and binary data.      The standards should be adaptable to different cryptographic algorithms.      The standards should work along with other XML security standards and specifications.

Various links that enable table operations have been put together on one sub-page of the Table view: Operations. Here is an overview of this sub-page: Table Maintenance During the lifetime of a table, it repeatedly gets modified, and so grows and shrinks. Outages may occur on the server, leaving some tables in a damaged state. Using the Operations sub-page, we can perform various operations, but not every operation is available for every table type: Check table: Scans all rows to verify that deleted links are correct. Also, a checksum is calculated to verify the integrity of the keys; we should get an 'OK' message if everything is all right. Analyze table: Analyzes and stores the key distribution; this will be used on subsequent JOIN operations to determine the order in which the tables should be joined. Repair table: Repairs any corrupted data for tables in the MyISAM and ARCHIVE engines. Note that the table might be so corrupted that we cannot even go into Table view for it! In such a case, refer to the Multi-Table Operations section for the procedure to repair it. Optimize table: This is useful when the table contains overheads. After massive deletions of rows or length changes for VARCHAR fields, lost bytes remain in the table. PhpMyAdmin warns us in various places (for example, in the Structure view) if it feels the table should be optimized. This operation is a kind of defragmentation for the table. In MySQL 4.x, this operation works only on tables in the MyISAM, Berkeley (BDB), and InnoDB engines. In MySQL 5.x, it works only on tables in the MyISAM, InnoDB, andARCHIVE engines. Flush table: This must be done when there have been lots of connection errors and the MySQL server blocks further connections. Flushing will clear some internal caches and allow normal operations to resume. Defragment table: Random insertions or deletions in an InnoDB table fragment its index. The table should be periodically defragmented for faster data retrieval. The operations are based on the underlying MySQL queries available—phpMyAdmin is only calling those queries. Changing Table Attributes Table attributes are the various properties of a table. This section discusses the settings for some of them. Table Type The first attribute we can change is called Table storage engine: This controls the whole behavior of the table: its location (on-disk or in-memory), the index structure, and whether it supports transactions and foreign keys. The drop-down list may vary depending on the table types supported by our MySQL server. Changing the table type may be a long operation if the number of rows is large. Table Comments This allows us to enter comments for the table. These comments will be shown at appropriate places (for example, in the left panel, next to the table name in the Table view and in the export file). Here is what the left panel looks like when the $cfg['ShowTooltip'] parameter is set to its default value of TRUE: The default value of $cfg['ShowTooltipAliasDB'] and $cfg['ShowTooltipAliasTB'] (FALSE) produces the behavior we have seen earlier: the true database and table names are displayed in the left panel and in the Database view for the Structure sub-page. Comments appear when the mouse pointer is moved over a table name. If one of these parameters is set toTRUE, the corresponding item (database names for DB and table names for TB) will be shown as the tooltip instead of the names. This time, the mouse-over shows the true name for the item. This is convenient when the real table names are not meaningful. There is another possibility for $cfg['ShowTooltipAliasTB']: the 'nested' value. Here is what happens if we use this feature: The true table name is displayed in the left panel. The table comment (for example project__) is interpreted as the project name and is displayed as such. Table Order When we Browse a table or execute a statement such as SELECT * from book, without specifying a sort order, MySQL uses the order in which the rows are physically stored. This table order can be changed with the Alter table order by dialog. We can choose any field, and the table will be reordered once on this field. We choose author_id in the example, and after we click Go, the table gets sorted on this field. Reordering is convenient if we know that we will be retrieving rows in this order most of the time. Moreover, if later we use an ORDER BY clause and the table is already physically sorted on this field, the performance should be higher. This default ordering will last as long as there are no changes in the table (no insertions, deletions, or updates). This is why phpMyAdmin shows the (singly) warning. After the sort has been done on author_id, books for author 1 will be displayed first, followed by the books for author 2, and so on. (We are talking about a default browsing of the table without explicit sorting.) We can also specify the sort order: Ascending or Descending. If we insert another row, describing a new book from author 1, and then click Browse, the book will not be displayed along with the other books for this author because the sort was done before the insertion. Table Options Other attributes that influence the table's behavior may be specified using the Table options dialog: The options are: pack_keys:Setting this attribute results in a smaller index; this can be read faster but takes more time to update. Available for the MyISAM storage engine. checksum: This makes MySQL compute a checksum for each row. This results in slower updates, but easier finding of corrupted tables. Available for MyISAM only. delay_key_write: This instructs MySQL not to write the index updates immediately but to queue them for later, which improves performance. Available for MyISAM only. auto-increment: This changes the auto-increment value. It is shown only if the table's primary key has the auto-increment attribute. Renaming, Moving, and Copying Tables The Rename operation is the easiest to understand: the table simply changes its name and stays in the same database. The Move operation (shown in the following screen) can manipulate a table in two ways: change its name and also the database in which it is stored Moving a table is not directly supported by MySQL, so phpMyAdmin has to create the table in the target database, copy the data, and then finally drop the source table. The Copy operation leaves the original table intact and copies its structure or data (or both) to another table, possibly in another database. Here, the book-copy table will be an exact copy of the book source table. After the copy, we will stay in the Table view for the book table unless we selected Switch to copied table. The Structure only copy is done to create a test table with the same structure. Appending Data to a Table The Copy dialog may also be used to append (add) data from one table to another. Both tables must have the same structure. This operation is achieved by entering the table to which we want to copy the data of the current table and choosing Data only. For example, we would want to append data when book data comes from various sources (various publishers), is stored in more than one table, and we want to aggregate all the data to one place. For MyISAM, a similar result can be obtained by using the MERGE storage engine (which is a collection of identical MyISAM tables), but if the table is InnoDB, we need to rely on phpMyAdmin's Copy feature. Multi-Table Operations In the Database view, there is a checkbox next to each table name and a drop-down menu under the table list. This enables us to quickly choose some tables and perform an operation on all those tables at once. Here we select the book-copy and the book tables, and choose the Check operation for these tables. We could also quickly select or deselect all the checkboxes with Check All/Uncheck All. Repairing an "in use" Table The multi-table mode is the only method (unless we know the exact SQL query to type) for repairing a corrupted table. Such tables may be shown with the in use flag in the database list. Users seeking help in the support forums for phpMyAdmin often receive this tip from experienced phpMyAdmin users. Database Operations The Operations tab in the Database view gives access to a panel that enables us to perform operations on a database taken as a whole. Renaming a Database Starting with phpMyAdmin 2.6.0, a Rename database dialog is available. Although this operation is not directly supported by MySQL, phpMyAdmin does it indirectly by creating a new database, renaming each table (thus sending it to the new database), and dropping the original database. Copying a Database Since phpMyAdmin 2.6.1, it is possible to do a complete copy of a database, even if MySQL itself does not support this operation natively. Summary In this article, we covered the operations we can perform on whole tables or databases. We also took a look at table maintenance operations for table repair and optimization, changing various table attributes, table movements, including renaming and moving to another database, and multi-table operations.

Introduction Welcome to the first in a new series of articles on Photoshop - the Photoshop Foundations series. The aim of this series is to give both beginners and more experienced users all the information they need to use Photoshop as efficiently as possible. Photoshop is a huge application, and there is usually more than one way to look at a given subject, or perform a certain action. This series aims to both, guide you through the more confusing aspects of Photoshop and show you the very best ways to use this application. In this first article we are going to look at the difference between vector and bitmap graphics, which is one of the most important principles to understand when working with graphics on a computer, inside or outside of Photoshop. Although Photoshop primarily is a bitmap image editor, it is capable of handling vector graphics to a certain extent. This can be a little confusing for people new to creating graphics on a computer, but by the end of this article you should have a clear idea of the difference between these two types of graphics. Bitmap Graphics Bitmap graphics are made up of colored pixels. Pixels are very small rectangles (usually square, although in some video applications they are wider than they are tall) of varying colors that once put together give you an image. You can see from the example below that zooming in on a bitmap image reveals the pixels that make up the image when viewed at 100%.   Bitmap graphics are usually (but not always) photographic in nature, capable of subtle graduated tones - often in the range of millions of colors per image. The problem with bitmap graphics is that they don't enlarge well as Photoshop needs to guess what color the extra pixels should be - this can result is loss of definition and a dramatic lowering in quality, depending on how much you enlarge the image. Common file formats for bitmap image data include GIF, JPEG and PNG for Internet usage and TIFF for print usage. As you can see from the example below, physically enlarging an image will degrade quality. Pixels are also used to display the image on your computer screen. Common pixel dimensions of computer displays are 1024 wide by 768 high and 1600 wide by 1200 high. The size of a bitmap graphic when viewed on your computer screen is defined by the number of pixels that make up the image - so an image that is 50 pixels wide will look very small on your screen at 100% viewing percentage, whereas an image that is 4000 pixels wide will be larger than your screen at 100% viewing percentage. The printable dimensions of an image are defined by the DPI (dots per inch) - this information is invisibly embedded in the image file. Digital cameras often embed information such as this, that may include the conditions the image was taken in, and even the camera model used. This information is not actually visible in the image, and requires software such as Photoshop to read it. You should not confuse the output DPI of your printer with this figure, which may range from 600-2400DPI - this refers to the density of the dots of ink laid down on the page by the printer. You don't have to prepare your images to 2400 DPI to get the best results - in fact doing so will significantly slow down printing as your file could potentially be huge! Often an image DPI in the range of 175-250 will give very good results on home printers. Images prepared for high quality commercial print are usually prepared at 300 DPI for up to A3 in size; whereas very large images (for instance on billboards) can be as low as 50 DPI, as they are not made to be viewed as closely as a magazine or small poster. There is no need to go above 300 DPI when creating images as you will yield virtually no improvement in output quality, only increasing the size of your file when saved. It is easy to understand the relationship between pixel dimensions and DPI - put simply, the DPI is how many pixels will be printed in an inch - so you could actually think of DPI as PPI (pixels per inch). Indeed, many experts believe this to be the true definition of DPI, and that Photoshop should refer to it as such. However, the term DPI is used throughout the professional print industry, so this is why it is referred to as DPI in Photoshop, not PPI.

The Core Database Much of the data we see in Joomla! is stored in the database. A base installation has over thirty tables. Some of these are related to core extensions and others to the inner workings of Joomla!. There is an official database schema, which describes the tables created during the installation. For more information, please refer to: http://dev.joomla.org/ component/option,com_jd-wiki/Itemid,31/id,guidelines:database/. A tabular description is available at: http://dev.joomla.org/downloads/Joomla15_DB-Schema.htm. We access the Joomla! database using the global JDatabase object. The JDatabase class is an abstract class, which is extended by different database drivers. There are currently only two database drivers included in the Joomla! core, MySQL and MySQLi. We access the global JDatabase object using JFactory: $db =& JFactory::getDBO(); Extending the Database When we create extensions, we generally want to store data in some form. If we are using the database, it is important to extend it in the correct way. Table Prefix All database tables have a prefix, normally jos_, which helps in using a single database for multiple Joomla! installations. When we write SQL queries, to accommodate the variable table prefix, we use a symbolic prefix that is substituted with the actual prefix at run time. Normally the symbolic prefix is #__, but we can specify an alternative prefix if we want to. Schema Conventions When we create tables for our extensions, we must follow some standard conventions. The most important of these is the name of the table. All tables must use the table prefix and should start with name of the extension. If the table is storing a specific entity, add the plural of the entity name to the end of the table name separated by an underscore. For example, an items table for the extension 'My Extension' would be called #__myExtension_items. Table field names should all be lowercase and use underscore word separators; you should avoid using underscores if they are not necessary. For example, you can name an email address field as email. If you had a primary and a secondary email field, you could call them email and email_secondary; there is no reason to name the primary email address email_primary. If you are using a primary key record ID, you should call the field id, make it of type integer auto_increment, and disallow null. Doing this will allow you to use the Joomla! framework more effectively. Common Fields We may use some common fields in our tables. Using these fields will enable us to take advantage of the Joomla! framework. Publishing We use publishing to determine whether to display data. Joomla! uses a special field called published, of type tinyint(1); 0 = not published, 1 = published. Hits If we want to keep track of the number of times a record has been viewed, we canuse the special field hits, of type integer and with the default value 0. Checking Out To prevent more than one user trying to edit one record at a time we can check out records (a form of software record locking). We use two fields to do this, checked_out and checked_out_time. checked_out, of type integer, holds the ID of the user that has checked out the record. checked_out_time, of type datetime, holds the date and time when the record was checked out. A null date and a user ID of 0 is recorded if the record is not checked out. Ordering We often want to allow administrators the ability to choose the order in which items appear. The ordering field, of type integer, can be used to number records sequentially to determine the order in which they are displayed. This field does not need to be unique and can be used in conjunction with WHERE clauses to form ordering groups. Parameter Fields We use a parameter field, a TEXT field normally named params, to store additional information about records; this is often used to store data that determines how a record will be displayed. The data held in these fields is encoded as INI strings (which we handle using the JParameter class). Before using a parameter field, we should carefully consider the data we intend to store in the field. Data should only be stored in a parameter field if all of the following criteria are true: Not used for sorting records Not used in searches Only exists for some records Not part of a database relationship Schema Example Imagine we have an extension called 'My Extension' and an entity called foobar. The name of the table is #__myextension_foobars. This schema describes the table: Field Datatype NOT NULL AUTO INC UNSIGNED DEFAULT id INTEGER X X X NULL content TEXT X       checked_out INTEGER X   X 0 checked_out_time DATETIME X     0000-00-00 00:00:00 params TEXT X       ordering INTEGER X   X 0 hits INTEGER X   X 0 published TINYINT(1) X   X 0 This table uses all of the common fields and uses an auto-incrementing primary keyID field. When we come to define our own tables we must ensure that we use thecorrect data types and NOT NULL, AUTO INC, UNSIGNED and DEFAULT values. The SQL displayed below will create the table described in the above schema: CREATE TABLE `#__myextension_foobars` ( `id` INTEGER UNSIGNED NOT NULL DEFAULT NULL AUTO_INCREMENT, `content` TEXT NOT NULL DEFAULT '', `checked_out` INTEGER UNSIGNED NOT NULL DEFAULT 0, `checked_out_time` DATETIME NOT NULL DEFAULT '0000-00-00 00:00:00', `params` TEXT NOT NULL DEFAULT '', `ordering` INTEGER UNSIGNED NOT NULL DEFAULT 0, `hits` INTEGER UNSIGNED NOT NULL DEFAULT 0, `published` INTEGER UNSIGNED NOT NULL DEFAULT 0, PRIMARY KEY(`id`)) CHARACTER SET `utf8` COLLATE `utf8_general_ci`; Date Fields We regularly use datetime fields to record the date and time at which an action has taken place. When we use these fields, it is important that we are aware of the effect of time zones. All dates and times should be recorded in UTC+0 (GMT / Z). When we come to display dates and times we can use the JDate class. The JDate class allows us to easily parse dates, output them in different formats, and apply UTC time-zone offsets. For more information about time zones, please refer to http://www.timeanddate.com. We often use parsers before we display data to make the data safe or to apply formatting to the data. We need to be careful how we store data that is going to be parsed. If the data is ever going to be edited, we must store the data in its RAW state. If the data is going to be edited extremely rarely and if the parsing is reversible, we may want to consider building a 'reverse-parser'. This way we can store the data in its parsed format, eradicating the need for parsing when we view the data and reducing the load on the server. Another option available tous is to store the data in both formats. This way we only have to parse data when we save it. Dealing with Multilingual Requirements Unlike ASCII and ANSII, Unicode is a multi-byte character set; it uses more than eight bits (one byte) per character. When we use UTF-8 encoding, character byte lengths vary. Unfortunately, MySQL versions prior to 4.1.2 assume that characters are always eight bits (one byte), which poses some problems. To combat the issue when installing extensions we have the ability to define different SQL files for servers, that do and do not support UTF-8. In MySQL servers that do not support UTF-8, when we create fields, which define a character length, we are actually defining the length in bytes. Therefore, if we try to store UTF-8 characters that are longer than one byte, we may exceed the size of the field. To combat this, we increase the length of fields to try to accommodate UTF-8strings. For example, a varchar(20) field becomes a varchar(60) field. We triple the size of fields because, although UTF-8 characters can be more than three bytes, the majority of common characters are a maximum of three bytes. This poses another issue, if we use a varchar(100) field, scaling it up for a MySQL server, which does not support UTF-8, we would have to define it as a varchar(300) field. We cannot do this because varchar fields have a maximum size of 255. The next step is slightly more drastic. We must redefine the field type so as it will accommodate at least three hundred bytes. Therefore, a varchar(100) field becomes a text field. As an example, the core #__content table includes a field named title. For MySQL severs that support UTF-8, the field is defined as: `title` varchar(255) NOT NULL default '' For MySQL severs that do not support UTF-8, the field is defined as: `title` text NOT NULL default '' We should also be aware that using a version of MySQL that does not support UTF-8 would affect the MySQL string handling functions. For example ordering by a string field may yield unexpected results. While we can overcome this using postprocessing in our scripts using the JString class, the recommended resolution is to upgrade to the latest version of MySQL.

The Basic Calendar Class The most basic type of calendar is the single-panel Calendar which is created with the YAHOO.widget.Calendar class. To display a calendar, an HTML element is required to act as a container for the calendar. The screenshot shows a basic Calendar control: The constructor can then be called specifying, at the very least the id of the container element as an argument. You can also specify the id of the Calendar object as an argument, as well as an optional third argument that can accept a literal object containing various configuration properties. The configuration object is defined within curly braces within the class constructor. It contains a range of configuration properties and keys that can be used to control different Calendar attributes such as its title, a comma-delimited range of pre-selected dates, or a close button shown on the calendar. There are a large number of methods defined in the basic Calendar class; some of these are private methods that are used internally by the Calendar object to do certain things and which you normally wouldn't need to use yourself. Some of the more useful public methods include: Initialization methods including init, initEvents, and initStyles which initialize either the calendar itself or the built-in custom events and style constants of the calendar. A method for determining whether a date is outside of the current month: isDateOOM. Navigation methods such as nextMonth, nextYear, previousMonth, and previousYear that can be used to programmatically change the month or year displayed in the current panel. Operational methods such as addMonths, addYears, subtractMonths, and subtractYears which are used to change the month and year shown in the current panel by the specified number of months or years. The render method is used to draw the calendar on the page and is called in for every implementation of a calendar, after it has been configured. Without this method, no Calendar appears on the page. Two reset methods: reset which resets the Calendar to the month and year originally selected, and resetRenderers which resets the render stack of the calendar. Selection methods that select or deselect dates such as deselect, deselectAll, desellectCell, select, and selectCell. As you can see, there are many methods that you can call to take advantage of the advanced features of the calendar control. The CalendarGroup Class In addition to the basic calendar, you can also create a grouped calendar that displays two or more month panels at once using the YAHOO.widget.CalendarGroup class. The control automatically adjusts the Calendar's UI so that the navigation arrows are only displayed on the first and last calendar panels, and so that each panel has its own heading indicating which month it refers to. The CalendarGroup class contains additional built-in functionality for updating the calendar panels on display, automatically. If you have a two-panel calendar displaying, for example, January and February, clicking the right navigation arrow will move February to the left of the panel so that March will display as the right-hand panel. All of this is automatic and nothing needs to be configured by you. There are fewer methods in this class; some of those found in the basic Calendar class can also be found here, such as the navigation methods, selection methods, and some of the render methods. Native methods found only in the CalendarGroup class include: The subscribing methods sub and unsub, which subscribe or unsubscribe to custom events of each child calendar. Child functions such as the callChildFunction and setChildFunction methods which set and call functions within all child calendars in the calendar group. Implementing a Calendar To complete this example, the only tool other than the Yahoo User Interface (YUI) that you'll need is a basic text editor. Native support for the YUI is provided by some web authoring software packages, most notably Aptana, an open-source application that has been dubbed 'Dreamweaver Killer'. However, I always find that writing code manually while learning something is much more beneficial. It is very quick and easy to add the calendar, as the basic default implementations require very little configuration. It can be especially useful in forms where the visitor must enter a date. Checking that a date has been entered correctly and in the correct format takes valuable processing time, but using the YUI calendar means that dates are always exactly as you expect them to be. So far we've spent most of this article looking at a lot of the theoretical issues surrounding the library; I don't know about you, but I think it's definitely time to get on with some actual coding! The Initial HTML Page Our first example page contains a simple text field and an image, which once clicked will display the Calendar control on the page, thereby allowing for a date to be selected and added to the input. Begin with the following basic HTML page: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd"><html lang="en"><head><meta http-equiv="content-type" content="text/html;charset=utf-8"><title>YUI Calendar Control Example</title><link rel="stylesheet"type="text/css"href="yui/build/calendar/assets/skins/sam/calendar.css"><script type="text/javascript"src="yui/build/yahoo-dom-event/yahoo-dom-event.js"></script><script type="text/javascript"src="yui/build/calendar/calendar-min.js"></script><style type="text/css">input {margin:0px 10px 0px 10px;}</style></head><body class="yui-skin-sam"><div><label>Please enter your date of birth:</label><input type="text" name="dobfield" id="dobfield"><img id="calico" src="icons/cal.png"alt="Open the Calendar control"></div><div id="mycal"></div></body></html> We begin with a valid DOCTYPE declaration, a must in any web page. For validity, we can also add the lang attribute to the opening <html> tag and for good measure, enforce the utf-8 character set. Nothing so far is YUI-specific, but coding in this way every time is a good habit. We link to the stylesheet used to control the appearance of the calendar control, which is handled in this example by the sam skin within the <link> tag. Accordingly, we also need to add the appropriate class name to the <body> tag. Following this, we link to the required library files with <script> tags; the calendar control is relatively simple and requires just the YAHOO, Document Object Model (DOM), and Event components (using the aggregated yahoo-dom-event.js file for efficiency), as well as the underlying source file calendar-min.js. A brief <style> tag finishes the <head> section of the page with some CSS relevant to this particular example, and the <body> of the page at this stage contains just two <div> elements: the first holds a <label>, text field, and a calendar icon (which can be used to launch the control), while the second holds the calendar control. When viewed in a browser, the page at this point should appear like this: The calendar icon used in this example was taken, with gratitude from Mark Carson at http://markcarson.com. Beginning the Scripting We want the calendar to appear when the icon next to the text field is clicked, rather than it being displayed on the page-load, so the first thing we need to do is to set a listener for the click event on the image. Directly before closing </body> tag, add the following code: <script type="text/javascript">//create the namespace object for this exampleYAHOO.namespace("yuibook.calendar");//define the lauchCal function which creates the calendarYAHOO.yuibook.calendar.launchCal = function() {}//create calendar on page load YAHOO.util.Event.onDOMReady(YAHOO.yuibook.calendar.launchCal);</script> Let's look at each line of the above code. We first use the .namespace() method of the YAHOO utility to set up the namespace object used for this example. Next we define the anonymous launchCal function, which will hold all of the code that generates the calendar control. Then we use the .onDOMReady() method of the Event utility to execute the launchCal function when the DOM is in an usable state. We'll be looking at the DOM utility in much greater detail later in the book. Now we can add the extremely brief code that's required to actually produce the Calendar. Within the braces of our anonymous function, add the following code: //create the calendar object, specifying the containerVar myCal = new YAHOO.widget.Calendar("mycal");//draw the calendar on screenmyCal.render();//hide it again straight awaymyCal.hide(); This is all that we need to create the Calendar; we simply define myCal as a new Calendar object, specifying the underlying container HTML element as an argument of the constructor. Once we have a Calendar object, we can call the .render() method on it to create the calendar and display it on the page. No arguments are required for this method. Since we want the calendar to be displayed when its icon is clicked, we hide the calendar from view straight away. To display the calendar when the icon for it is clicked, we'll need one more anonymous function. Add the following code beneath the .hide() method: //define the showCal function which shows the calendarVar showCal = function() {//show the calendarmyCal.show();} Now we can attach a listener which detects the click event on the calendar icon: //attach listener for click event on calendar iconYAHOO.util.Event.addListener("calico", "click", showCal); Save the file that we've just created as calendar.html or similar in your yuisite directory. If you view it in your browser now and click the Calendar icon, you should see this: The calendar is automatically configured to display the current date, although this is something that can be changed using the configuration object mentioned earlier. If you use a DOM explorer to view the current DOM of a page with an open calendar on it, you'll see that a basic Calendar control is rendered as a table with eight rows and seven columns. The first row contains the images used to navigate between previous or forthcoming months and the title of the current month and year. The next row holds the two-letter representations of each of the different days of the week, and the rest of the rows hold the squares representing the individual days of the current month. The screenshot on the next page show some of the DOM representation of the Calendar control used in our example page: Now that we can call up the Calendar control by clicking on our Calendar icon, we need to customize it slightly. Unless the person completing the form is very young, they will need to navigate through a large number of calendar pages in order to find their date of birth. This is where the Calendar Navigator interface comes into play. We can easily enable this feature using a configuration object passed into the Calendar constructor. Alter your code so that it appears as follows: //create the calendar object, using container & config objectmyCal = new YAHOO.widget.Calendar("mycal", {navigator:true}); Clicking on the Month or Year label will now open an interface which allows your visitors to navigate directly to any given month and year: The configuration object can be used to set a range of calendar configuration properties including the original month and year displayed by the Calendar, the minimum and maximum dates available to the calendar, a title for the calendar, a close button, and various other properties.

Third-Party Video Providers Many sites desiring video will choose to use a third-party video provider such as YouTube or Blip.TV. This reduces the bandwidth requirement from their server, is easy to include in their posts, and allows videos to be easily shared virtually by users across the Internet. The easiest way, without further configuration of a basic Drupal installation, for an administrator to include a third-party video is to simply paste the video's embedded code in a post. Most video providers will offer a snippet of HTML that may be copied from a particular video page, which will embed the video. However, this requires using a filter that will allow <object>, <embed>, and <param> tags. But since they open the door to attacks on the site, they should only be used by administrators and trusted editors. You could also use the Full HTML filter, but this is even more dangerous as allowing that filter to be generally used would open the site to cross-site scripting (XSS) attacks. First, you'll need to set up a filter that allows the tags. Add an input format at Administer | Site configuration | Input formats (at /admin/settings/filters/add). After naming the filter, check the role(s) you wish to give access to this filter such as edit role. Check HTML corrector, HTML filter, and Line break converter. After pressing Save configuration, click on the Configure tab. Using YouTube as an example, an administrator would first need to upload a video to YouTube. This will require an account at YouTube, but they make it fairly painless for a user to jump in and contribute videos. You'll just need to follow their instructions: Once you have a video there, you will find the embedded code on the video page. You will need to click in the text field where that is provided, and copy the HTML for pasting on your own page: Next you will submit a node on your site such as from Node | Add | Page (at /node/add/page), and paste the embed code in the body for the node. You will need to enable either the new filter created earlier or Full HTML, as the embedded code will contain object and/or embedded tags, which would be filtered out by the default filter in Drupal. If you want editors to have the ability to select their filter, you will need to enable that ability for a role, and possibly set up a new filter depending on your needs. Also note that you will need to disable the TinyMCE Rich Text Editor when embedding video directly into content if the TinyMCE module is enabled on your site. After submitting, your video will appear in the content. As with any HTML embedded in your node body, you may manually place your video at any point within the content such as after the second paragraph or at the end of the node: Embedded Embedded Media Field Finally, we come to the alternative of hosting video from our own servers. Although using a module such as Media Mover combined with services such as Amazon S3 makes serving video a slightly easier task than it might have otherwise been, for most sites the bandwidth required for serving video is generally not a viable option. Additionally, sites may wish to take advantage of the viral opportunities of hosting video through a widely recognized provider such as YouTube or Blip.TV. There are several modules that provide some limited support for embedding third-party media, including both the Video and Asset modules. However, at the time of this writing, the most comprehensive and by far the easiest to configure and use is the Embedded Media Field, which includes the Embedded Video Field as part of its package. Install both of these modules and set up a new content type with an Embedded Video Field. You will need, of course, to have the CCK (Content) module installed as well. As with our other examples, you will first add your type from Administer | Content management | Content types | Add content type (at /admin/content/types/add), give it a name such as Video, and add the field from Administer | Content management | Content types | Video | Add field (at /admin/content/node-type/video/add_field). Before continuing, I must confess a bias here. I wrote the original Embedded Media Field module with assistance from Sam Tresler during DrupalCamp NYC in 2007, and rewrote it for a more solid and flexible API during OSCMS later that year. I am also indebted to Alex Urevick-Ackelsberg for his assistance in the ongoing maintenance and support. Without doing anything else, you may now add a new video from a provider by simply pasting its URL into the field. The module will then automatically parse and display the video appropriately. There are several settings on the following page that may be set, including allowed providers, video and thumbnail sizes, and whether the video plays automatically. You may leave the providers alone to allow content from any of them, or select only the providers you wish to allow editors and users to use The local checkbox is experimental at the time of this writing and may not actually be on the version you're reading. The module maintainers (myself included, of course) intend to hook into other APIs to provide better local video support without reinventing the wheel. That may or may not be ready by the time you read this book. The Custom URL provider is also used to experimentally support direct videos from any source available from an HTTP request, including your local server. It is not recommended for general use, as it would be easy to use that to unethically hotlink to videos from someone else's server. Hundreds of flying monkeys will hunt you down if you do that. Basically, always turn off support for that unless you have a specific (and moral) use for that feature. You can set video sizes in the next sections for full size and preview size video display. By default, videos will be displayed in full size. You can change the display to video preview or thumbnail at the display settings page, by browsing to Administer | Content management | Content types | Video | Display fields (at /admin/content/node-type/video/display). Videos will be forced to display at the size provided here, regardless of how they are offered by the provider. You can also determine if the video will autoplay or not. For instance, you might use a small video preview for teasers and a larger full-size video when viewing the node page, turning on the autoplay in that case. Finally, you may wish to use thumbnails, for instance when displaying a video as a teaser or when using views. Note that thumbnails are not yet supported for all video providers. Some providers do not offer an easy API to discover a particular video's thumbnail file. To learn if thumbnails are supported by a particular provider, go to Administer | Content management | Embedded Media Field Configuration (at /admin/content/emfield) and open the fieldset for Embedded Video Field. You will see the supported features for each provider within their particular fieldsets, where you may also disable them or enable unique settings. You may wish to provide for custom thumbnails, whether for providers lacking an automatic thumbnail or for any external video in general. For this purpose, the Embedded Custom Thumbnail module is included in the module's package. Just enable that module, and then check the Allow custom thumbnails for this field box on the type's administration screen We now have a full-featured video field in place, which is as easy to use as cut and paste. Summary Video is still a maturing media on the Internet. Much has happened as it has exploded onto sites across the world, and contributors to Drupal have made recent strides in supporting it. However, there is still much to be done to make it easier for administrators to support it. Also, although there are many new and traditional tools available such as Views and Embedded Media Field, these still require some setup to get working.

(For more resources on Microsoft, see here.) Initial State of Computer Assuming you are working with the Windows XP OS, it will be advisable to create a restore point to which you can fall back should you fail to install and something goes wrong. You can set up a fall back position by going to Start | All Programs | Accessories | System Tools | System Restore. This allows you to comeback where you were before starting the install. The other thing that you should lookup is the suite of Microsoft software you already have on your computer that may interfere with the product you are installing. This can be reviewed following Start | Control Panel | Add and Remove Programs. SQL 2008 server requires IE 6.0 or higher version. It may be helpful to install this before embarking on installing the SQL 2008 Server. For the purpose of this article IE 7.0 was installed. It has appeared in some forum topics that SQL 2008 can exist side-by-side with SQL 2005 server. However in the present case SQL 2005 was completely removed. Sometimes even this removal is not quite an easy process if something is broken in the original install and requires you to reinstall and then uninstall. In the case of SQL Server 2008, there was an earlier version, "Katmai", installed but never used due to its inability to connect to the SQL Server Management Studio (Well, unless you remove the SQL 2005 client you cannot install SQL 2008 Client), a fact which came to light much later. 'Katmai' components were completely removed which required reinstalling the 'Katmai' followed by its complete removal. When you download the SQL 2008 and run the executable, it creates the folder, servers, containing a number of subfolders and files (dynamic link library files etc) that are used during the installation. Help can be accessed from servershelp1033s10ch_setup, an HTML file which provides a wealth of information regarding all aspects of installation including migration from an earlier version. From servesdefault.htm you can begin the installation which provides the required support using Prepare | Install | Other information navigational aid. After removing all the suggested components during this installation, the remaining Microsoft SQL Server related components on the computer are as shown in the Add and Remove Programs window in the next figure. The very first screen you will see when you click on the serverssetup.exe file is the SQL Server 2008 Setup where you need to agree with the licensing terms before proceeding. When you click on the Next button which displays the Installation Pre-requisites screen, you will be shown the pending items needed before you install SQL 2008 server. Click on the Install button after highlighting the pending item regarding setup support files in the right screen. SQL Server Installation Center This will take you to the SQL Server Installation Center screen as shown. It has a number of useful hyperlinks that you can come back to by repeating the above steps. Click on New Installation link. This Starts Install SQL Server 2008 Wizard for System Configuration Check. After a while when the checking is completed the following screen will be displayed. This timeall items have the status marked 'Passed'. In a previous attempt when the 'Katmai' items were still uninstalled,the Previous CTP Install Check did not succeed and it was corrected only after completely removing those items. Clicking on Next button takes you to screen where you need to select the features that you want to have installed as shown. The display shows Features Selection window after all items have been checked.

Creating an Entity Data Model You can create the ADO.NET Entity Data Model in one of the two ways: Use the ADO.NET Entity Data Model Designer Use the command line Entity Data Model Designer called EdmGen.exe We will first take a look at how we can design an Entity Data Model using the ADO.NET Entity Data Model Designer which is a Visual Studio wizard that is enabled after you install ADO.NET Entity Framework and its tools. It provides a graphical interface that you can use to generate an Entity Data Model. Creating the Payroll Entity Data Model using the ADO.NET Entity Data Model Designer Here are the tables of the 'Payroll' database that we will use to generate the data model: Employee Designation Department Salary ProvidentFund To create an entity data model using the ADO.NET Entity Data Model Designer, follow these simple steps: Open Visual Studio.NET and create a solution for a new web application project as seen below and save with a name. Switch to the Solution Explorer, right click and click on Add New Item as seen in the following screenshot: Next, select ADO.NET Entity Data Model from the list of the templates displayed as shown in the following screenshot:   Name the Entity Data Model PayrollModel and click on Add. Select Generate from database from the Entity Data Model Wizard as shown in the following screenshot: Note that you can also use the Empty model template to create the Entity Data Model yourself. If you select the Empty Data Model template and click on next, the following screen appears: As you can see from the above figure, you can use this template to create the Entity Data Model yourself. You can create the Entity Types and their relationships manually by dragging items from the toolbox. We will not use this template in our discussion here. So, let's get to the next step. Click on Next in the Entity Data Model Wizard window shown earlier. The modal dialog box will now appear and prompts you to choose your connection as shown in the following figure: Click on New Connection Now you will need to specify the connection properties and parameters as shown in the following figure: We will use a dot to specify the database server name. This implies that we will be using the database server of the localhost, which is the current system in use. After you specify the necessary user name, password, and the server name, you can test your connection using the Test Connection button. When you do so, the message Test connection succeeded gets displayed in the message box as shown in the previous figure. When you click on OK on the Test connection dialog box, the following screen appears: <connectionStrings> <add name="PayrollEntities" connectionString="metadata=res:// *; provider=System.Data.SqlClient;provider connection string=&quot; Data Source=.;Initial Catalog=Payroll;User ID=sa;Password=joydip1@3; MultipleActiveResultSets=True&quot;" providerName="System.Data.EntityClient" /> </connectionStrings> Note the Entity Connection String generated automatically. This connection string will be saved in the ConnectionStrings section of your application's web.config file. This is how it will look like: When you click on Next in the previous figure, the following screen appears: Expand the Tables node and specify the database objects that you require in the Entity Data Model to be generated as shown in the following figure: Click on Finish to generate the Entity Data Model. Here is the output displayed in the Output Window while the Entity Data Model is being generated: Your Entity Data Model has been generated and saved in a file named PayrollModel.edmx. We are done creating our first Entity Data Model using the ADO.NET Entity Data Model Designer tool. When you open the Payroll Entity Data Model that we just created in the designer view, it will appear as shown in the following figure: Note how the Entity Types in the above model are related to one another. These relationships have been generated automatically by the Entity Data Model Designer based on the relationships between the tables of the Payroll database. In the next section, we will learn how we can create an Entity Data Model using the EdmGen.exe command line tool. Creating the Payroll Data Model Using the EdmGen Tool We will now take a look at how to create a data model using the Entity Data Model generation tool called EdmGen. The EdmGen.exe command line tool can be used to do one or more of the following: Generate the .cdsl, .msl, and .ssdl files as part of the Entity Data Model Generate object classes from a .csdl file Validate an Entity Data Model The EdmGen.exe command line tool generates the Entity Data Model as a set of three files: .csdl, .msl, and .ssdl. If you have used the ADO.NET Entity Data Model Designer to generate your Entity Data Model, the .edmx file generated will contain the CSDL, MSL, and the SSDL sections. You will have a single .edmx file that bundles all of these sections into it. On the other hand, if you use the EdmGen.exe tool to generate the Entity Data Model, you would find three distinctly separate files with .csdl, .msl or .ssdl extensions. Here is a list of the major options of the EdmGen.exe command line tool: Option Description /help Use this option to display help on all the possible options of this tool. The short form is /? /language:CSharp Use this option to generate code using C# language /language:VB Use this option to generate code using VB language /provider:<string> Use this option to specify the name of the ADO.NET data provider that you would like to use. /connectionstring: <connection string> Use this option to specify the connection string to be used to connect to the database /namespace:<string> Use this option to specify the name of the namespace /mode:FullGeneration Use this option to generate your CSDL, MSL, and SSDL objects from the database schema /mode:EntityClassGeneration Use this option to generate your entity classes from a given CSDL file /mode:FromSsdlGeneration Use this option to generate MSL, CSDL, and Entity Classes from a given SSDL file /mode:ValidateArtifacts Use this option to validate the CSDL, SSDL, and MSL files /mode:ViewGeneration Use this option to generate mapping views from the CSDL, SSDL, and MSL files  

JMeter is a 100% pure Java desktop application. JMeter is found to be very useful and convenient in support of functional testing. Although JMeter is known more as a performance testing tool, functional testing elements can be integrated within the Test Plan, which was originally designed to support load testing. Many other load-testing tools provide little or none of this feature, restricting themselves to performance-testing purposes. Besides integrating functional-testing elements along with load-testing elements in the Test Plan, you can also create a Test Plan that runs these exclusively. In other words, aside from creating a Load Test Plan, JMeter also allows you to create a Functional Test Plan. This flexibility is certainly resource-efficient for the testing project. In this article by Emily H. Halili, we will give you a walkthrough on how to create a Test Plan as we incorporate and/or configure JMeter elements to support functional testing. Preparing for Functional Testing JMeter does not have a built-in browser, unlike many functional-test tools. It tests on the protocol layer, not the client layer (i.e. JavaScripts, applets, and many more.) and it does not render the page for viewing. Although, by default that embedded resources can be downloaded, rendering these in the Listener | View Results Tree may not yield a 100% browser-like rendering. In fact, it may not be able to render large HTML files at all. This makes it difficult to test the GUI of an application under testing. However, to compensate for these shortcomings, JMeter allows the tester to create assertions based on the tags and text of the page as the HTML file is received by the client. With some knowledge of HTML tags, you can test and verify any elements as you would expect them in the browser. It is unnecessary to select a specific workload time to perform a functional test. In fact, the application you want to test may even reside locally, with your own machine acting as the "localhost" server for your web application. For this article, we will limit ourselves to selected functional aspects of the page that we seek to verify or assert. Using JMeter Components We will create a Test Plan in order to demonstrate how we can configure the Test Plan to include functional testing capabilities. The modified Test Plan will include these scenarios: 1. Create Account —New Visitor creating an Account 2. Login User —User logging in to an Account Following these scenarios, we will simulate various entries and form submission as a request to a page is made, while checking the correct page response to these user entries. We will add assertions to the samples following these scenarios to verify the 'correctness' of a requested page. In this manner, we can see if the pages responded correctly to invalid data. For example, we would like to check that the page responded with the correct warning message when a user enters an invalid password, or whether a request returns the correct page. First of all, we will create a series of test cases following the various user actions in each scenario. The test cases may be designed as follows: CREATE ACCOUNT Test Steps Data Expected 1 Go to Home page. www.packtpub.com Home page loads and renders with no page error 2 Click Your Account link (top right). User action 1. Your Account page loads and renders with no page error.2. Logout link is not found. 3 No Password: - Enter email address in Email text field.- Click the Create Account and Continue button. email=EMAIL 1. Your Account page resets with Warning message-Please enter password.2. Logout link not found. 4 Short Password: - Enter email address in Email text field.- Enter password in Password text field.- Enter password in Confirm Password text field. - Click Create Account and Continue button. email=EMAILpassword=SHORT_PWD confirm password=SHORT_PWD 1. Your Account page resets with Warning message-Your password must be 8 characters or longer.2. Logout link is not found. 5 Unconfirmed Password: - Enter email address in Email text field.- Enter password in Password text field.- Enter password in Confirm Password text field. - Click Create Account and Continue button. email=EMAILpassword=VALID_PWDconfirm password=INVALID_PWD 1. Your Account page resets with Warning messagePassword does not match.2. Logout link is not found. 6 Register Valid User: - Enter email address in Email text field.- Enter password in Password text field.- Enter password in Confirm Password text field. - Click Create Account and Continue button. email=EMAILpassword=VALID_PWDconfirm password=VALID_PWD 1. Logout link is found.2. Page redirects to User Account page.3. Message found: You are registered as: e:<EMAIL>. 7 Click Logout link. User action 1. Logout link is NOT found.     LOGIN USER Test Steps Data Expected 1 Click Home page. User action 1. WELCOME tab is active. 2 Log in Wrong Password: - Enter email in Email text field- Enter password at Password text field.- Click Login button. email=EMAILpassword=INVALID_PWD 1. Logout link is NOT found.2. Page refreshes.3. Warning message-Sorry your password was incorrect appears. 3 Log in Non-Exist Account:- Enter email in Email text field.- Enter password in Password text field.- Click Login button. email=INVALID_EMAILpassword=INVALID_PWD 1. Logout link is NOT found.2. Page refreshes.3. Warning message-Sorry, this does not match any existing accounts. Please check your details and try again or open a new account below appears. 4 Log in Valid Account:- Enter email in Email text field.- Enter password in Password text field.- Click Login-button. email=EMAILpassword=VALID_PWD 1. Logout link is found.2. Page reloads.3. Login successful message-You are logged in as: appears. 5 Click Logout link. User action 1. Logout link is NOT found.    

Categories, Products, and Attributes Products are the items that are sold. In Magento, Categories organize your Products, and Attributes describe them. Think of a Category as the place where a Product lives, and an Attribute is anything that describes a Product. Each one of your Products can belong to one or more Categories. Also, each Product can be described by any number of Attributes. Is it a Category or an Attribute? Some things are clearly Categories. For example, if you have an electronics store, MP3 players would make a good Category. If you're selling jewellery, earrings would make a good Category. Other things are clearly Attributes. Color, description, picture, and SKU number are almost always Attributes. Sometimes, the same thing can be used for a Category or an Attribute. For example, suppose your site sells shoes. If you made size an Attribute, then after your shoppers have located a specific shoe, they can select the size they want. However, if you also made size a Category, the shoppers could begin their shopping by selecting their size. Then they could browse through the styles available in their size. So should size be an Attribute, a Category, or both? The answer depends upon what kind of shopping experience you want to create for your customers. Examples The hierarchy of Categories, Products, and Attributes looks like this: Category 1 Product 1 Attribute 1 Attribute 2 Product 2 Attribute 1 Attribute 2 Category 2 Product 3 Attribute 1 Attribute 3 Product 4 Attribute 1 Attribute 3 We are building a site that sells gourmet coffee, so we might organize our store like this: Single Origin Hawaiian Kona Grind (whole bean, drip, French press) Roast (light, medium, dark) Blue Mountain Grind Roast Blends Breakfast Blend Grind Caffeine (regular, decaffeinated) Afternoon Cruise Grind Caffeine In Magento, you can give your shoppers the ability to search your store. So if the shoppers know that they want Blue Mountain coffee, they can use the Search function to find it in our store. However, customers who don't know exactly what they want will browse the store. They will often begin browsing by selecting a category. With the organization that we just saw, when customers browse our store, they will start by selecting Single Origin or Blends. Then the shoppers will select the product they want: Hawaiian Kona, Blue Mountain, Breakfast Blend, or Afternoon Cruise. After our shoppers decide upon a Product, they select Attributes for that product. In our store, shoppers can select Grind for any of the products. For Single Origin, they can also select Roast. For blends, they can select Caffeine. This gives you a clue about how Magento handles attributes. To each Product, you can apply as many, or as few, attributes as you want. Now that we have definitions for Category, Product, and Attribute, let's look at each of them in detail. Then, we can start adding products. Categories Product Categories are important because they are the primary tool that your shoppers use to navigate your store. Product Categories organize your store for your shoppers. Categories can be organized into Parent Categories and Subcategories. To get to a Subcategory, you drill down through its Parent Category. Categories and the Navigation Menu If a Category is an Anchor Category, then it appears on the Navigation Menu. The term "Anchor" makes the category sound as if it must be a top-level category. This is not true. You can designate any category as an Anchor Category. Doing so puts that category into the Navigation Menu. When a shopper selects a normal Category from the Navigation Menu, its landing page and any subcategories are displayed. When a shopper selects an Anchor Category from the menu, Magento does not display the normal list of subcategories. Instead, it displays the Attributes of all the Products in that category and its subcategories. Instead of moving down into subcategories, the shopper uses the Attributes to filter all the Products in that Anchor Category and the Categories below it. The Navigation Menu will not display if: You don't create any Categories, or You create Categories, but you don't make any of them Anchors, or Your Anchor Categories are not subcategories under the Default Category. The Navigation Menu will display only if: You have created at least one Category You have made at least one Category an Anchor You have made the Anchor Category a Subcategory under Default. When you first create your Magento site and add Products, you won't see those Products on your site until you've met all of the previous conditions. For this reason I recommend that you create at least one Anchor Category before you start adding Products to your store. As you add each Product, add it to an Anchor Category. Then, the Product will display in your store, and you can preview it. If the Anchor Category is not the one that you want for that Product, you can change the Product's Category later Before we add Products to our coffee store, we will create two Anchor Categories: Single Origin and Blends. As we add Products, we will assign them to a Category so that we can preview them in our storefront. Making best use of Categories There are three things that Categories can accomplish. They can: Help the shoppers, who know exactly what they want, to find the product that they are looking for. Help the shoppers, who almost know what they want, to find a product that matches their desires. Entice the shoppers, who have almost no idea of what they want, to explore your store. We would like to organize our store so that our Categories accomplish all these goals. However, these goals are often mutually exclusive. For example, suppose you create an electronics store. In addition to many other products, your store sells MP3 players, including Apple iPods. A Category called iPods would help the shoppers who know that they want an iPod, as they can quickly find one. However, the iPods Category doesn't do much to help shoppers who know that they want an MP3 player, but don't know what kind. On the Web, you usually search something when you know what you want. But when you're not sure about what you want, you usually browse. In an online store, you usually begin browsing by selecting a Category. When you are creating Categories for your online store, try to make them helpful for shoppers who almost know what they want. However, what if a high percentage of your shoppers are looking for a narrow category of products? Consider creating a top-level Category to make those products easily accessible. Again, suppose you have an electronics store that sells a wide variety of items. If a high percentage of your customers want iPods, it might be worthwhile to create a Category just for those few products. The logs from the Search function on your site are one way you can determine whether your shoppers are interested in a narrow Category of a Product. Are 30 percent of the searches on your site for left-handed fishing reels? If so, you might want to create a top-level Category just for those Products. Attributes An Attribute is a characteristic of a Product. Name, price, SKU, size, color, and manufacturer are all examples of Attributes. System versus Simple Attributes Notice that the first few examples (name, price, and SKU) are all required for a Product to function in Magento. Magento adds these Attributes to every product, and requires you to assign a value for each of them. These are called System Attributes. The other three examples (size, color, and manufacturer) are optional Attributes. They are created by the store owner. They are called Simple Attributes. When we discuss creating and assigning Attributes, we are almost always discussing Simple Attributes. Attribute Sets Notice that the Single Origin coffees have two Attributes: Grind and Roast. Also notice that the blends have the Attributes of Grind and Caffeine. Single Origin Hawaiian Kona Grind (whole bean, drip, French press) Roast (light, medium, dark) Blue Mountain Grind Roast Blends Breakfast Blend Grind Caffeine (regular, decaffeinated) Afternoon Cruise Grind Caffeine In this example, the store owner created three Attributes: Grind, Roast, and Caffeine. Next, the store owner grouped the Attributes into two Attribute Sets: one set contains Grind and Roast, and the other set contains Grind and Caffeine. Then, an Attribute set was applied to each Product. Attributes are not applied directly to Products. They are first grouped into Attribute Sets, and then a set can be applied to a Product. This means that you will need to create a set for each different combination of Attributes in your store. You can name these Sets after the Attributes they contain, such as Grind-Roast. Or, you can name them after the type of Product which will use those Attributes, such as Single Origin Attributes. If each Product in a group will use the same Attribute as every other Product in that group, then you can name the set after that group. For example, at this time, all Single Origin coffees have the same Attributes: Grind and Roast. If they will all have these two Attributes and you will always add and remove Attributes to them as a group, then you could name the set Single Origin Attributes. If the Products in a group will likely use different Attributes, then name the set after the Attributes. For example, if you expect that some Single Origin coffees will use the Attributes Grind and Roast, while others will use just Roast, then it would not make sense to create a set called Single Origin Attributes. Instead, create a set called Grind-Roast, and another called Roast. Three types of Products In Magento, you can create three different types of Products: Simple, Configurable, and Grouped. The following is a very brief definition for each type of Product. Simple Product A Simple Product is a single Product, with Attributes that the store owner chooses. As the saying goes, "What you see is what you get." The customer does not get to choose anything about the Product. In our coffee store, a good example for a Simple Product might be a drip coffee maker. It comes in only one color. And while the customer can buy drip coffee makers of various sizes (4 cups, 8 cups, 12 cups, and so on), each of those is a separate Product. A bad example of a Simple Product would be a type of coffee. For example, we might want to allow the customer to choose the type of roast for our Hawaiian Kona coffee: light, medium, or dark. Because we want the customer to choose a value for an Attribute, that would not be a good Simple Product. Configurable Product A Configurable Product is a single Product, with at least one Attribute that the customer gets to choose. There is a saying that goes, "Have it your way." The customer gets to choose something about the Product. A good example of a Configurable Product would be a type of coffee that comes in several different roasts: light, medium, and dark. Because we want the customer to choose the roast(s) he wants, that would be a good Configurable Product. Grouped Product A Grouped Product is several Simple Products that are displayed on the same page. You can force the customer to buy the group, or allow the customer to buy each Product separately. The previous definitions are adequate for now. However, when you start creating Products, you will need to know more about each type of Product.

Confidentiality and integrity are two critical components of web services. While confidentiality can be ensured by means of encryption, the encrypted data can still be overwritten and the integrity of the message can be compromised. So it becomes is equally important to protect the integrity of the message; digital signatures helps us in doing just that. Overview of Digital Signatures In the web services scenario, XML messages are exchanged between the client application and the web services. Certain messages contain critical business information, and therefore the integrity of the message should be ensured. Ensuring the integrity of the message is not a new concept, it has been there for a long time. The concept is to make sure that the data was not tampered while in transit between the sender and the receiver. Consider, for example, that Alice and Bob are exchanging emails that are critical to business. Alice wants to make sure that Bob receives the correct email that she sent and no one else tampered with or modified the email in between. In order to ensure the integrity of the message, Alice digitally signs the message using her private key, and when Bob receives the message, he will check to make sure that the signature is still valid before he can trust or read the email. What is this digital signature? And how does it prove that no one else tampered with the data? When a message is digitally signed, it basically follows these steps: Create a digest value of the message(a unique string value for the message using a SHA1 or MD5 algorithm). Encrypt the digest value using the private key—known only to the sender. Exchange the message along with the encrypted digest value. MD5 and SHA1 are message digest algorithms to calculate the digest value. The digest or hash value is nothing but a non-reversible unique string for any given data, i.e. the digest value will change even if a space is added or removed. SHA1 produces a 160 bit digest value, while MD5 produces a 128 bit value. When Bob receives the message, his first task is to validate the signature. Validation of signature goes through a sequence of steps: Create a digest value of the message again using the same algorithm. Encrypt the digest value using the public key of Alice(obtained out of band or part of message, etc.) Validate to make sure that the digest value encrypted using the public key matches the one that was sent by Alice. Since the public key is known or exchanged along with the message, Bob can check the validity of the certificate itself. Digital certificates are issued by a trusted party such as Verisign. When a certificate is compromised, you can cancel the certificate, which will invalidate the public key. Once the signature is verified, Bob can trust that the message was not tampered with by anyone else. He can also validate the certificate to make sure that it is not expired or revoked, and also to ensure that no one actually tampered with the private key  of Alice. Digital Signatures in Web Services In the last section, we learnt about digital signatures. Since web services are all about interoperability, digital-signature-related information is represented in an industry standard format called XML Signature (standardized by W3C). The following are the key data elements that are represented in an interoperable manner by XML Signature: What data (what part of SOAP message) is digitally signed? What hash algorithm (MD5 or SHA1) is used to create the digest value? What signature algorithm is used? Information about the certificate or key. In the next section, we will describe how the Oracle Web Services Manager can help generate and verify signatures in web services. Signature Generation Using Oracle WSM Oracle Web Services Manager can centrally manage the security policy, including digital signature generation. One of the greatest advantages in using Oracle WSM to digitally sign messages is that the policy information and the digital certificate information are centrally stored and managed. An organization can have many web services, and some of them might exchange certain business critical information and require that the messages be digitally signed. Oracle WSM will play a key role when different web services have different requirements to sign the message, or when it is required to take certain actions before or after signing the message. Oracle WSM can be used to configure the signature at each web service level and that reduces the burden of deploying certificates across multiple systems. In this section, we will discuss more about how to digitally sign the response message of the web service using Oracle WSM. Sign Message Policy Step As a quick refresher, in Oracle WSM, each web service is registered within a gateway or an agent and a policy is attached to each web service. The policy steps are divided mainly into request pipeline template and response pipeline template, where different policies can be applied for request or response message processing. In this section, I will describe how to configure the policy for a response pipeline template to digitally sign the response message. It is assumed that the web service is registered within a gateway and a detailed example will be described later in this article . In the response pipeline, we can add a policy step called Sign Message to digitally sign the message. In order to digitally sign a message, the key components that are required are: Private key store Private key password The part of SOAP message that is being signed The signature algorithm being used The following screenshot describes the Sign Message policy step with certain values populated.   In the previous screenshot, the values that are populated are: Keystore location—The location where the private key file is located. Keystore type—Whether or not it is PKCS12 or JKS. Keystore password—The password to the keystore. Signer's private-key alias—The alias to gain access to the private key from the keystore. Signer's private-key password—The password to access the private key. Signed Content—Whether the BODY or envelope of the SOAP message should be signed. The above information is a part of a policy that is attached to the time service which will sign the response message. As per the information that is shown in the screenshot, the BODY of the SOAP message response will be digitally signed us in the SHA1 as the digest algorithm, and PKCS12 key store. Once the message is signed, the SOAP message will look like: <?xml version="1.0" encoding="UTF-8"?><soap:Envelope soap_encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" > <soap:Header> <wsse:Security soap_mustUnderstand="1"> <wsse:BinarySecurityToken ValueType="http://docs. oasis-open.org/wss/2004/01/oasis-200401-wss- x509-token-profile-1.0#X509v3" EncodingType="http://docs.oasis-open. org/wss/2004/01/oasis-200401-wss-soap-message- security-1.0#Base64Binary" wsu_Id="_ VLL9yEsi09I9f5ihwae2lQ22" >SecurityTOkenoKE2ZA==< /wsse:BinarySecurityToken> <dsig:Signature > <dsig:SignedInfo> <dsig:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/ xml-exc-c14n#"/> <dsig:SignatureMethod Algorithm="http://www.w3.org/2000/09/ xmldsig#rsa-sha1"/> <dsig:Reference URI="#ishUwYWW2AAthrx hlpv1CA22"> <dsig:Transforms> <dsig:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/> </dsig:Transforms> <dsig:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/> <dsig:DigestValue>ynuqANuYM3qzdTnGOLT7SMxWHY=</dsig:DigestValue> </dsig:Reference> <dsig:Reference URI="#UljvWiL8yjedImz 6zy0pHQ22"> <dsig:Transforms> <dsig:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/> </dsig:Transforms> <dsig:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/> <dsig:DigestValue>9ZebvrbVYLiPv1BaVLDaLJVhwo=</dsig:DigestValue> </dsig:Reference> </dsig:SignedInfo> <dsig:SignatureValue>QqmUUZDLNeLpAEFXndiBLk</dsig:SignatureValue> <dsig:KeyInfo> <wsse:SecurityTokenReference wsu_Id="_7vjdWs1ABULkiLeE7Y4lAg22" > <wsse:Reference URI="#_VLL9yEsi09I9f5ihwae2lQ22"/> </wsse:SecurityTokenReference> </dsig:KeyInfo> </dsig:Signature> <wsu:Timestamp wsu_Id="UljvWiL8yjedImz6zy0pHQ22"> <wsu:Created>2007-11-16T15:13:48Z</wsu:Created> </wsu:Timestamp> </wsse:Security> </soap:Header> <soap:Body wsu_Id="ishUwYWW2AAthrxhlpv1CA22" > <n:getTimeResponse > <Result xsi_type="xsd:string">10:13 AM</Result> </n:getTimeResponse> </soap:Body></soap:Envelope>

Procedural Textures vs. Bitmap Textures Blender has basically two types of textures, which are procedural textures and bitmap textures. Each one has both positive and negative points. Which one is the best will depend on your project needs. Procedural: This kind of texture is generated by the software at rendering time, just like vector lines. This means that it won't depend on any type of image file. The best thing about this type of texture is that it is resolution independent, so we can set the texture to be rendered with high resolutions with minimum loss of quality. The negative point about this kind of texture is that it's harder to get realistic textures with it. Bitmap: To use this kind of texture, we will need an image file, such as a JPEG, PNG, or TGA file. The good thing about these textures is that we can achieve very realistic materials with it quickly. On the other hand, we must find the texture file before using it. And there is more. If you are creating a high resolution render, the texture file must be big. Texture Library Do you remember the way we organized materials? We can do exactly the same thing about textures. Besides setting names and storing the Blender files to import and use again later, collecting bitmap textures is another important point. Even if you don't start right away, it's important to know where to look for textures. So here is a small list of websites that provides free texture download. http://www.blender-textures.org http://www.cgtextures.com http://blender-archi.tuxfamily.org/textures Applying Textures To use a texture, we must apply a material to an object, and then use the texture with this material. We always use the texture inside a material. For instance, to make a plane that simulates a marble floor, we have to use a texture and set up how the surface will react to light and texture, which can give the surface a proper look of marble using any texture. To do that, we must use the texture panel, which is located right next to the materials button. We can use a keyboard shortcut to open this panel: just hit F6. There is a way to add a texture in the material panel as well, with a menu called Texture. The best way to get all the options is to add the texture on the texture panel. On this panel, we will be able to see a lot of buttons, which represent the texture channels. Each one of these channels can hold a texture. The final texture will be a mix of all the channels. If we have a texture at channel 1 and another texture at channel 2, these textures will be blended and represented in the material. Before adding a new texture, we must select a channel by clicking over one of them. Usually the first channel is selected, but if you want to use another one, just click on the channel. When the channel is selected, just click the Add New button to add a new texture. The texture controls are very similar to the material controls. We can set a name for the texture at the top, or erase it if we don't want it anymore. With the selector, we can choose a previously created texture too—just click and select. Now comes the fun part. Having added a texture, we have to choose a texture type. To do that, we click on the texture type combo box. There are a lot of textures, but most of them are procedural textures and we won't use them much. The only texture type that isn't procedural is the image type. We can use textures like Clouds and Wood to create some effects and give surfaces a more complex look, or even create a grass texture with some dirt on it. But most times, the texture type that we will be using will be the Image type. Each texture has its own set of parameters to determine how it will look in the object. If we add a Wood texture, it will show the configuration parameters at the right. If we choose as texture type Clouds, the parameters showed at the right will be completely different. With the image texture type it's not different, this kind of texture has its own type of setup. This is the control panel: To show how to set up a texture, let's use an image file that represents a wood floor and a plane. We can apply the texture to this plane and set up how it's going to look, testing all the parameters. The first thing to do is assign a material to the plane, and add a texture to this material. We choose as texture type the Image option. It will show the configuration options for this kind of texture. To apply the image as a texture to the plane, just click on the Load button, situated on the Image menu. When we hit this button, we will be able to select the image file. Locate the image file and the texture will be applied. If we want to have more control over how this texture is organized and placed on the plane, we need to learn how the controls work. Every time you make any changes to the setup of a texture, these changes will be shown in the preview window; use it a lot to make good changes. Here is a list of what some of the buttons can do for the texture: UseAlpha: If the texture has an alpha channel, we have to press this button for Blender calculate the channel. An image has an alpha channel when some kind of transparency is stored in the image. For instance, a .png file with transparent background has an alpha channel. We can use this to create a texture with a logo, for a bottle, or to add an image of a tree or person to a plane. Rot90: With this option we can rotate the texture by 90 degrees. Repeat: Every texture must be distributed on the object surface, and repeating the texture in lines and columns is the default way to do that. Extended: If this button is pressed, the texture will be adjusted to fit all the object surface area. Clip: With this option, the texture will be cropped and we will be able to show only a part of it. To adjust which parts of the texture will be displayed, use the Min/Max X/Y options. Xrepeat / Yrepeat: This option determines how many times a texture is repeated, with the repeat option turned on. Normal Map: If the texture will be used to create Normal Maps, press this button. These are textures used to change the face normals of an object. Still: With this button selected, we can determine that the image used as texture is a still image. This option is marked by default. Movie: If you have to use a movie file as texture, press this button. This is very useful if we need to make something like a theatre projection screen or a tv screen. Sequence: We can use a sequence of images as texture too; just press this button. It works the same ways as with a movie file. There are a few more parameters, like the Reload button. If your texture file suffers any kind of change, we must press this button for the changes get accepted by Blender. The X button can erase this texture; use it if you need to select another image file. When we add a texture to any material, an external link is created with this file. This link can be absolute or relative. When we add a texture called "wood.png", which is located at the root of your main hard disk, like C:, a link to this texture will be created like this: "c:wood.png", so every time you open this file, the software will look for that file at that exact place. This is an absolute link, but we can use a relative link as well. For instance, when we add a texture located in the same folder as our scene, a relative link will be created. Every time we use an absolute link and we have to move the ".blend" file to another computer, the texture file must go with it. To imbue the image file with the .blend, just press the icon of gift package. To save all the textures used in a scene, just access the file menu and use the Pack Data option. It will make all the texture files embedded with the source blend file. Mapping Every time we add a texture to any object, we must choose a mapping type to set up how the texture will be applied to the object. For instance, if we have a wall and apply a wood texture, it must be placed like wallpaper. But for cylindrical or spherical objects, or even walls, we have to set up in a way that makes the texture adaptable to the topology of the surface, to avoid effects such as a stretched texture. To set this up, we use the mapping options, which are located on the Map Input menu. On this menu, we can choose between four basic mapping types which are Cube, Sphere, Flat, and Tube. If you have a wall, choose the option that matches the topology type with the model. In this case, the best choice is the Cube. Another important option here is the UV button, which allows us to use another very powerful type of texturing, based on UV Mapping. Normal Map This is a special and useful type of texture, that can change the normals of surfaces. If we have a floor and a texture of ceramic tiles, the surface can be represented with smaller details of that tiling, using this kind of a map. It's almost like modeling the tiles. But everything is created using just a normal map. To use this kind of texture, we must turn on the Nor button on the Map To menu. When this button is turned on, we can set up the Nor slider to determine the intensity of the normal displacement. It works based on the pixel color of the texture. With white pixels, the normals are not affected, and with black pixels, the normals are fully translated. If you want to optimize the normal mapping, using a special texture is much recommended. Some texture libraries even have this type of normal maps ready for use. They can be called bump maps too. Here is an example of how we can use them. We take a stone texture and a tiled texture with a white background and black lines. The stone texture is applied to the floor, and the tiled texture is used to create a tiling for the floor. The setup for that is really simple. Just apply the texture at a lower channel, and turn off the Col button for this channel. Turn on the Nor button, and this texture will affect only the normals and not the material color. Any image can be used as a normal map, but we will always get better results with a greyscale image prepared to be used as a normal map. Now, just set up the Nor intensity with the slider, and see the render. Turn on positive and turn on negativeSome of the buttons on the Map To menu can be turned on with positive and negative values. For instance, the Nor option can be turned on with one click. If we click on it again, the Nor text will turn yellow. This means that the Nor is inverted with negative values. Some other buttons may present the same option.