Android Application Testing: Getting Started


Android Application Testing Guide

Android Application Testing Guide Build intensively tested and bug free Android applications
        Read more about this book      

(For more resources related to this subject, see here.)

We will avoid introductions to Android and the Open Handset Alliance ( as they are covered in many books already and I am inclined to believe that if you are reading this article covering this more advanced topic you have started with Android development before.

However, we will be reviewing the main concepts behind testing and the techniques, frameworks, and tools available to deploy your testing strategy on Android.

Brief history

Initially, when Android was introduced by the end of 2007, there was very little support for testing in the platform, and for some of us very accustomed to using testing as a component intimately coupled with the development process, it was the time to start developing some frameworks and tools to permit this approach.

By that time Android had some rudimentary support for unit testing using JUnit (, but it was not fully supported and even less documented.

In the process of writing my own library and tools, I discovered Phil Smith's Positron (originally at and now renamed and moved to, an Open Source library and a very suitable alternative to support testing on Android, so I decided to extend his excellent work and bring some new and missing pieces to the table.

Some aspects of test automation were not included and I started a complementary project to fill that gap, it was consequently named Electron. And although positron is the anti-particle of the electron, and they annihilate if collide, take for granted that that was not the idea, but more the conservation of energy and the generation of some visible light and waves.

Later on, Electron entered the first Android Development Challenge (ADC1) in early 2008 and though it obtained a rather good score in some categories, frameworks had no place in that competition. Should you be interested in the origin of testing on Android, please find some articles and videos that were published in my personal blog (

By that time Unit Tests could be run on Eclipse. However, testing was not done on the real target but on a JVM on the local development computer.

Google also provided application instrumentation code through the Instrumentation class. When running an application with instrumentation turned on, this class is instantiated for you before any of the application code, allowing you to monitor all of the interaction the system has with the application. An Instrumentation implementation is described to the system through an AndroidManifest.xml file.

Software bugs

It doesn't matter how hard you try and how much time you invest in design and even how careful you are when programming, mistakes are inevitable and bugs will appear.

Bugs and software development are intimately related. However, the term bugs to describe flaws, mistakes, or errors has been used in hardware engineering many decades before even computers were invented. Notwithstanding the story about the term bug coined by Mark II operators at Harvard University, Thomas Edison wrote this in 1878 in a letter to Puskás Tivadar showing the early adoption of the term:

"It has been just so in all of my inventions. The first step is an intuition, and comes with a burst, then difficulties arise — this thing gives out and [it is] then that 'Bugs' — as such little faults and difficulties are called — show themselves and months of intense watching, study and labor are requisite before commercial success or failure is certainly reached."

How bugs severely affect your projects

Bugs affect many aspects of your software development project and it is clearly understood that the sooner in the process you find and squash them, the better. It doesn't matter if you are developing a simple application to publish on the Android Market, you are re-branding the Android experience for an operator, or creating a customized version of Android for a device manufacturer, bugs will delay your shipment and will cost you money.

From all of the software development methodologies and techniques, Test Driven Development, an agile component of the software development process, is likely the one that forces you to face your bugs earlier in the development process and thus it is also likely that you will solve more problems up front.

Furthermore, the increase in productivity can be clearly appreciated in a project where a software development team uses this technique versus one that is, in the best of the cases, writing tests at the end of the development cycle. If you have been involved in software development for the mobile industry, you will have reasons to believe that with all the rush this stage never occurs. It's funny because usually, this rush is to solve problems that could have been avoided.

In a study conducted by the National Institute of Standards and Technology (USA) in 2002, it was reported that software bugs cost the country economy $59.5 billion annually. More than a third of this cost can be avoided if better software testing is performed.

But please, don't misunderstand this message. There are no silver bullets in software development and what will lead you to an increase in productivity and manageability of your project is the discipline applying these methodologies and techniques to stay in control.

Why, what, how, and when to test

You should understand that early bug detection saves huge amount of project resources and reduces software maintenance costs. This is the best known reason to write software tests for your development project. Increased productivity will soon be evident.

Additionally, writing the tests will give you a deeper understanding of the requirements and the problem to be solved. You will not be able to write tests for a piece of software you don't understand.

This is also the reason behind the approach of writing tests to clearly understand legacy or third party code and having the infrastructure to confidently change or update it.

The more the code is covered by your tests, the higher could be your expectations of discovering the hidden bugs.

If during this coverage analysis you find that some areas of your code are not exercised, additional tests should be added to cover this code as well.

This technique requires a special instrumented Android build to collect probe data and must be disabled for any release code because the impact on performance could severely affect application behavior.

To fill in this gap, enter EMMA (, an open-source toolkit for measuring and reporting Java code coverage, that can offline instrument classes for coverage. It supports various coverage types:

  • class
  • method
  • line
  • basic block

Coverage reports can also be obtained in different output formats. EMMA is supported up to some degree by the Android framework and it is possible to build an EMMA instrumented version of Android.

This screenshot shows how an EMMA code coverage report is displayed in the Eclipse editor, showing green lines when the code has been tested, provided the corresponding plugin is installed.

(Move the mouse over the image to enlarge it.)

Unfortunately, the plugin doesn't support Android tests yet, so right now you can use it for your JUnit tests only. Android coverage analysis report is only available through HTML.

Tests should be automated and you should run some or all tests every time you introduce a change or addition to your code in order to ensure that all the conditions that were met before are still met and that the new code satisfies the tests as expected.

This leads us to the introduction of Continuous Integration. It relies on the automation of tests and building processes.

If you don't use automated testing, it is practically impossible to adopt Continuous Integration as part of the development process and it is very difficult to ensure that changes would not break existing code.

What to test

Strictly speaking you should test every statement in your code but this also depends on different criteria and can be reduced to test the path of execution or just some methods. Usually there's no need to test something that can't be broken, for example it usually makes no sense to test getters and setters as you probably won't be testing the Java compiler on your own code and the compiler would have already performed its tests.

In addition to the functional areas you should test, there are some specific areas of Android applications that you should consider. We will be looking at these in the following sections.

Activity lifecycle events

You should test that your activities handle lifecycle events correctly.

If your activity should save its state during onPause() or onDestroy() events and later be able to restore it in onCreate(Bundle savedInstanceState), you should be able to reproduce and test all these conditions and verify that the state was correctly saved and restored.

Configuration-changed events should also be tested as some of these events cause the current Activity to be recreated, and you should test correct handling of the event and the newly created Activity preserves the previous state. Configuration changes are triggered even by rotation events, so you should test you application's ability to handle these situations.

Database and filesystem operations

Database and filesystem operations should be tested to ensure that they are handled correctly. These operations should be tested in isolation at the lower system level, at a higher level through ContentProviders, or from the application itself.

To test these components in isolation, Android provides some mock objects in the android.test.mock package.

Physical characteristics of the device

Much before delivering your application you should be sure that all of the different devices it can be run on are supported or at less you should detect the situation and take pertinent measures.

Among other characteristics of the devices, you may find that you should test:

  • Network capabilities
  • Screen densities
  • Screen resolutions
  • Screen sizes
  • Availability of sensors
  • Keyboard and other input devices
  • GPS
  • External storage

In this respect Android Virtual Devices play an important role because it is practically impossible to have access to all of the devices with all of the possible combinations of features but you can configure AVD for almost every situation. However, as it was mentioned before, leave your final tests for actual devices where the real users will run the application to understand its behavior.

        Read more about this book      

(For more resources related to this subject, see here.)

Types of tests

Depending on the testing method employed, testing can be implemented at any time in the development process. However, we will be promoting testing at an early stage of the development effort, even before the full set of requirements have been defined and the coding process has been started.

There are several types of tests depending on the object under testing. Independently of its type, a test should verify a condition and return the result of this evaluation as a single Boolean value indicating its success or failure.

Unit tests

Unit tests are software tests written by programmers to programmers in a programming language and they should isolate the component under tests and be able to test it in a repeatable way. That's why usually unit tests and mock objects are placed together. You use mock objects to isolate the unit from its dependencies, to monitor interactions, and also to be able to repeat the test any number of times. For example, if your test deletes some data from a database you probably don't want the data to be actually deleted and not found the next time the test is run.

JUnit is the de-facto standard for unit tests on Android. It's a simple open source framework for automating unit testing, originally written by Erich Gamma and Kent Beck.

Android (up to Android 2.3 Gingerbread) uses JUnit 3. This version doesn't use annotations and uses introspection to detect the tests.

A typical JUnit test would be something like this (the actual tests are highlighted):

* Android Application Testing Guide
package com.example.aatg.test;

import JUnit.framework.TestCase;

* @author diego
public class MyUnitTests extends TestCase {
private int mFixture;

* @param name test name
public MyUnitTests(String name) {

/* (non-Javadoc)
* @see JUnit.framework.TestCase#setUp()
protected void setUp() throws Exception {
mFixture = 1234;

/* (non-Javadoc)
* @see JUnit.framework.TestCase#tearDown()
protected void tearDown() throws Exception {

* Preconditions
public void testPreconditions() {

* Test method
public void testSomething() {
fail("Not implemented yet");

The following sections explain in detail the components that build up our test case.

The test fixture

A test fixture is the well known state defined as a baseline to run the tests and is shared by all the test's cases, and thus plays a fundamental role in the design of the tests.

Generally, it is implemented as a set of member variables and following Android conventions they will have names starting with m, for example mActivity. However, it can also contain external data, as specific entries in a database or some files present in the filesystem.

The setUp() method

This method is called to initialize the fixture.

Overriding it you have the opportunity to create objects and initialize fields that will be used by tests. It's worth noting that this setup occurs before every test.

The tearDown() method

This method is called to finalize the fixture.

Overriding it you can release resources taken by the initialization or tests. Again, this method is invoked after every test.

For example, you can release a database or a network connection here.

JUnit is designed in a way where the entire tree of test instances is built in one pass, and then the tests are executed in a second pass. Therefore, the test runner holds strong references to all Test instances for the duration of the test execution. This means that for very large and very long test runs with many Test instances, none of the tests may be garbage collected until the end of the entire test run. This is particularly important in Android and while testing on limited devices as some test may fail not because of an intrinsic failure but because of the amount of memory needed to run the application plus its tests exceeding the device limits.

Therefore, if you allocate external or limited resources in a test, such as Services or ContentProviders, you are responsible for freeing those resources. Explicitly setting an object to null in the tearDown() method, for example, allows it to be garbage collected before the end of the entire test run.

Test preconditions

Usually there's no way to test for preconditions as the tests are discovered using introspection and their order could vary. Anyway, it's customary to create a testPreconditions() method to test for preconditions. Though it is not assured that this test would be called in any specific order, it is a good practice to hold this and the preconditions together for organizational purposes.

The actual tests

All public void methods whose names start with test will be considered as a test. JUnit 3, as opposed to JUnit 4, doesn't use annotations to discover the tests but introspection to find their names. There are some annotations available on Android test framework like @SmallTest, @MediumTest, or @LargeTest, but they don't turn a simple method into a test but organize them in different categories. Ultimately you will have the ability to run tests for a single category using the test runner.

As a rule of thumb, name your tests in a descriptive way and use nouns and the condition being tested.

For example: testValues(), testConversionError(), testConversionToString() are all valid tests names.

Test for exceptions and wrong values instead of just testing positive cases.

During the execution of the test some conditions, side effects, or method returns should be compared against the expectations. To ease these operations, JUnit provides a full set of assert* methods to compare the expected results from the test to the actual results after running them throwing exceptions if conditions are not met. Then the test runner handles these exceptions and presents the results.

These methods, which are overloaded to support different arguments, include:

  • assertEquals()
  • assertFalse()
  • assertNotNull()
  • assertNotSame()
  • assertNull()
  • assertSame()
  • assertTrue()
  • fail()

In addition to these JUnit assert methods, Android extends Assert in two specialized classes providing additional tests:

  • MoreAsserts
  • ViewAsserts

Mock objects

Mock objects are mimic objects used instead of calling the real domain objects to enable testing units in isolation.

Generally, this is accomplished to assert that correct methods are called but they would also be of help, as mentioned, to isolate your tests from the surrounding universe and be able to run them independently and repeatably.

Android testing framework supports several mock objects that you will find very useful when writing your tests and you need to provide some dependencies to be able to compile the tests.

Several classes are provided by the Android testing framework in the android.test.mock package:

  • MockApplication
  • MockContentProvider
  • MockContentResolver
  • MockContext
  • MockCursor
  • MockDialogInterface
  • MockPackageManager
  • MockResources

Almost any component of the platform that could interact with your Activity can be created instantiating one of these classes.

However, they are not real implementations but stubs where every method generates an UnsupportedOperationException and that you can extend to create real mock objects.

UI tests

Finally, special consideration should be taken if your tests involve UI components. As you may have already known, only the main thread is allowed to alter the UI in Android. Thus a special annotation @UIThredTest is used to indicate that a particular test should be run on that thread and would have the ability to alter the UI. On the other hand, if you only want to run parts of your test on the UI thread, you may use Activity.runOnUiThread(Runnable r) method providing the corresponding Runnable containing testing instructions.

A helper class TouchUtils is also provided to aid in the UI test creation allowing the generation of events to send to the Views, such as:

  • click
  • drag
  • long click
  • scroll
  • tap
  • touch

By these means you can actually remote control you application from the tests.

Eclipse and other IDEs support

JUnit fully supported by Eclipse and the Android ADT plugin lets you create Android testing projects. Furthermore, you can run the tests and analyze the results without leaving the IDE.

This also provides a more subtle advantage; being able to run the tests from Eclipse allows you to debug the tests that are not behaving correctly.

In the screenshot, we can see how Eclipse runs 18 tests taking 20.008 seconds, where 0 Errors and 0 Failures were detected. The name of each test and its duration is also displayed. If they were a failure, the Failure Trace would show the related information.

Android Application Testing: Getting Started

Other IDEs like ItelliJ and Netbeans have plugins integrating Android development up to some degree; still they are not officially supported.

Even if you are not developing in an IDE, you can find support to run the tests with ant (check if you are not familiar with this tool). This setup is done by the android command using the subcommand create test-project as described by this help:

$ android --help create test-project

android [global options] create test-project [action options]

Global options:
-v --verbose Verbose mode: errors, warnings and informational
are printed.
-h --help Help on a specific command.
-s --silent Silent mode: only errors are printed out.

Action "create test-project":
Creates a new Android project for a test package.
-p --path The new project's directory [required]
-m --main Path to directory of the app under test, relative
to the
test project directory [required]
-n --name Project name

As indicated by the help you should provide at least the path to the project (--path) and the path to the main project or the project under test (--main).

Integration tests

Integration tests are designed to test the way individual components work jointly. Modules that have been unit tested independently are now combined together to test the integration.

Usually Android Activities require some integration to the system infrastructure to be able to run. They need the Activity lifecycle provided by the ActivityManager, and access to resources, filesystem, and databases.

The same criteria apply to other Android components like Services or ContentProviders that need to interact with other parts of the system to achieve their function.

In all these cases there are specialized tests provided by the Android testing framework that facilitates the creation of tests for these components.

Functional or acceptance tests

In agile software development, functional or acceptance tests are usually created by business and Quality Assurance (QA) people and expressed in a business domain language. These are high level tests to test the completeness and correctness of a user story or feature. They are created ideally through collaboration between business customers, business analysts, QA, testers, and developers. However the business customers (product owners) are the primary owners of these tests.

Some frameworks and tools can help in this field, most notably FitNesse (, which can be easily integrated, up to some point, into the Android development process and would let you create acceptance tests and check their results.

Also check Fit, and Slim (Simple List Invocation Method), as an alternative to Fit.

Android Application Testing: Getting Started

Lately, a new trend named Behavior Driven Development has gained some popularity and in a very brief description it can be understood as the evolution of Test Driven Development. It aims to provide a common vocabulary between business and technology people in order to increase mutual understanding.

Behavior Driven Development can be expressed as a framework of activities based on three principles (more information can be found at

  • Business and technology should refer to the same system in the same way
  • Any system should have an identified, verifiable value to the business
  • Upfront analysis, design, and planning all have a diminishing return

To apply these principles, usually business people are involved in writing test case scenarios in a high level language and use some tool, such as jbehave ( the following example, these scenarios are translated into code in a programming language that expresses the same test scenario.

Performance tests

Performance tests measure performance characteristics of the components in a repeatable way. If performance improvements are required by some part of the application, the best approach is to measure performance before and after some change is introduced.

As it is widely known, premature optimization does more harm than good, so it's better to clearly understand the impact of your changes on the overall performance.

The introduction of the Dalvik JIT compiler in Android 2.2 changed some optimization patterns that were widely used in Android development. Nowadays, every recommendation about performance improvements in the Android developer's site is backed up by performance tests.

System tests

The system is tested as a whole and the interaction between the components, software and hardware, is exercised. Normally, system tests include additional classes of tests like:

  • GUI tests
  • Smoke tests
  • Performance tests
  • Installation tests

Android testing framework

Android provides a very advanced testing framework extending the industry standard JUnit with specific features suitable to implement all of the testing strategies and types we mentioned before. In some cases, additional tools are needed but the integration of these tools is in most of the cases simple and straightforward.

Most relevant key features of the Android testing environment include:

  • Android extensions to the JUnit framework that provide access to Android system objects.
  • An instrumentation framework that lets tests control and examine the application.
  • Mock versions of commonly-used Android system objects.
  • Tools for running single tests or test suites, with or without instrumentation.
  • Support for managing tests and test projects in the ADT Plugin for Eclipse and at the command line.


The instrumentation framework is the foundation of the testing framework. Instrumentation controls the application under tests and permits the injection of mock components required by the application to run. For example, you can create mock Contexts before the application starts and let the application use it.

All the interaction of the application with the surrounding environment can be controlled using this approach. You can also isolate your application in a restricted environment to be able to predict the results forcing the values returned by some methods or mocking persistent and unchanged data for ContentProvider, databases, or even the filesystem content.

A Standard Android project has its tests in a correlated project that usually have the same project name but end with Test. Inside this Test project, the AndroidManifest.xml declares the Instrumentation.

As an illustrative example, assume your project has a manifest like this:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="
<application android:icon="@drawable/icon"
<activity android:name=".SampleActivity"
<action android:name="android.intent.action.MAIN" />
<category android:name=
"android.intent.category.LAUNCHER" />
<uses-sdk android:minSdkVersion="7" />

In this case, the correlated Test project will have the following AndroidManifest.xml:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="
android:versionCode="1" android:versionName="1.0">
<application android:icon="@drawable/icon"
<uses-library android:name="android.test.runner" />
<uses-sdk android:minSdkVersion="7" />
android:label="Sample Tests" />
<uses-permission android:name="
android.permission.INJECT_EVENTS" />

Here Instrumentation package is the same package of the main application with the .test suffix added.

Then the Instrumentation is declared specifying the target package and the test runner, in this case the default custom runner android.test.InstrumentationTestRunner.

Also notice that both, the application under test and the tests are Android applications with their corresponding APKs installed. Internally, they will be sharing the same process and thus have access to the same set of features.

When you run a test application, the Activity Manager ( uses the instrumentation framework to start and control the test runner, which in turn uses instrumentation to shut down any running instances of the main application, starts the test application, and then starts the main application in the same process. This allows various aspects of the test application to work directly with the main application.

Test targets

During the evolution of your development project your tests would be targeted to different devices. From simplicity, flexibility, and speed of testing on an emulator to the unavoidable final testing on the specific devices you are intending your application to be run on, you should be able to run on all of them.

There are also some intermediate cases like running your tests on a local JVM virtual machine on the development computer or on a Dalvik virtual machine or Activity, depending on the case.

Every case has its pros and cons, but the good news is that you have all of these alternatives available to run your tests.

The emulator is probably the most powerful target as you can modify almost every parameter from its configuration to simulate different conditions for your tests. Ultimately, your application should be able to handle all of these situations, so it's much better to discover the problems upfront than when the application has been delivered.

The real devices are a requirement for performance tests, as it is somewhat difficult to extrapolate performance measurements from a simulated device. You will have the real user experience only when using the real device. Rendering, scrolling, flinging, and other cases should be tested before delivering the application.


We have reviewed the main concepts behind testing in general and Android in particular. Having acquired this knowledge will let us start our journey and start exploiting the benefits of testing in our software development projects.

So far, we have visited the following subjects:

  • We reviewed the early stages of testing on Android and mentioned some of the frameworks that gave the origin to the current alternatives.
  • We briefly analyzed the reasons behind testing and the whys, whats, hows, and whens to test. Furthermore, from now on we will concentrate more on exploring the hows, now that we can assume you are convinced by the arguments exposed.
  • We enumerated the different and most common types of tests you would need in your projects, described some of the tools we can count on our testing toolbox, and provided an introductory example of a JUnit unit test to better understand what we are discussing about.

Also, we analyzed these techniques from the Android perspective and mentioned the use of Instrumentation to run our Android tests.

Further resources related to this subject:

You've been reading an excerpt of:

Android Application Testing Guide

Explore Title