Chapter 6. Testing a Cascading Application

Debugging, testing, and Quality Assurance (QA) processes are essential but often overlooked skills for the development of a robust system. As with many distributed software systems, Cascading can be difficult to debug, but using the techniques in this chapter, the mysteries of a Cascading application can be unlocked and understood. While there is little documentation on this topic, this chapter will provide in-depth information on how to efficiently test a Cascading application using many techniques.

Debugging a Cascading application

We will now discuss several different techniques and capabilities that will help you debug a Cascading application that you have built. These techniques will help you pinpoint bugs and errors in your code, aid in developing tests that can help to maintain correctness through a full development and maintenance life cycle, show you how to handle run-time errors, and also ensure that your application runs as fast as possible.

Getting your environment ready for debugging

Setting up your development environment is relatively straightforward, but there are a lot of details needed. Here we are going to discuss setting up Eclipse in order to be able to develop and test Cascading applications. We will look at doing this in Cascading local mode, and then will also discuss how to modify the environment to be able to produce JAR files that can be run against a Hadoop cluster. Lastly, we will discuss some of the techniques that are used to create a Hadoop environment that...

Testing strategies

In this section, we will discuss three test strategies. We will start with unit testing, then discuss integration testing, and finally load testing.

Unit testing involves creating very specific test cases that ensure that individual components are working as expected. In many cases, this will be at the custom operation level. Typically unit tests are coded by the developer who created the to-be-tested code. Unit tests are also important to check when code can be broken by maintenance or fixes. In many cases, the local mode is used for unit testing.

Integration testing is used when the individual components are assembled into a more complex function or full application. Integration testing is often also called end-to-end testing. It involves not only ensuring that all processing is correct, but it typically involves negative testing where erroneous conditions such as bad data must be handled.

Load testing is performed to test the overall scalability of the application. This...

Summary

In this chapter, we completed an in-depth review of many techniques used to debug and test a Cascading application. You learned how to set up Cascading developer configurations for rapid and thorough testing. We looked at several facilities and techniques for debugging, including the use of specialized filters such as Debug(), how to write log messages, and how to review the overall structure of your Cascading application visually. We also reviewed various testing strategies for unit, integration, and load/performance testing.

In the next chapter, we will take a look at some specific techniques that you can use to optimize the performance of your Cascading applications, and also how the performance and tuning of the underlying platform (that is, Hadoop) can be optimized to further assess and improve performance.