Getting Started with PhantomJS

By Aries Beltran
    Advance your knowledge in tech with a Packt subscription

  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Getting Started

About this book

PhantomJS is a headless WebKit browser with JavaScript API that allows you to create new ways to automate web testing. PhantomJS is currently being used by a large number of users to help them integrate headless web testing into their development processes. It also gives you developers a new framework to create web-based applications, from simple web manipulation to performance measurement and monitoring.

A step step-by by-step guide that will help you develop new tools for solving web and testing problems in an effective and quick way. The book will teach you how to use and maximize PhantomJS to develop new tools for web scrapping, web performance measurement and monitoring, and headless web testing. This book will help you understand PhantomJS’ scripting API capabilities and strengths.

This book starts by looking at PhantomJS’ JavaScript API, features, and basic execution of scripts. Throughout the book, you will learn details to help you write scripts to manipulate web documents and fully create a web scrapping tool.

Through its practical approach, this book strives to teach you by example, where each chapter focuses on the common and practical usage of PhantomJS, and how to extract meaningful information from the web and other services.

By the end of the book, you will have acquired the skills to enable you to use PhantomJS for web testing, as well as learning the basics of Jasmine, and how it can be used with PhantomJS.

Publication date:
November 2013
Publisher
Packt
Pages
140
ISBN
9781782164227

 

Chapter 1. Getting Started

PhantomJS is a new solution that provides headless testing of web applications. It is also a tool for dynamically capturing and rendering pages as images. It allows you to programmatically manipulate page content to control and change it to different forms. It can scrape websites and save important information to files. It will also provide you network-level information of your page and site resources. These are just a few of the functions that PhantomJS can do for us. It provides a fresh and a whole new way for web designers, testers, and developers to perform and create browser-based solutions.

PhantomJS uses QtWebKit as its core browser capability and uses the WebKit JavaScript engine for script interpretation and execution. Anything and everything that you can do in a WebKit-based browser (such as Chrome, Safari, and Opera browser) you can do with PhantomJS. It's more than just a browser because it supports web standards, such as CSS selector, DOM manipulation, JSON, HTML5 Canvas, and SVG; moreover, you can do some cool stuff such as performing file system I/O, accessing system environment variables, or even instantiating your own implementation of a web server daemon.

 

Downloading PhantomJS


Before we go through the features of PhantomJS, first we need to get our copy of the PhantomJS binaries. Typically, PhantomJS provides downloadable releases of binaries that are precompiled and packaged. You can choose from Linux, Mac OS X, and Windows precompiled packages. To download a copy, go to http://www.phantomjs.org/download.html.

Download your binaries based on your preference of operating system. After downloading, extract the binaries to any folder you desire. That's it! Your PhantomJS binary is ready to be used.

Tip

Add PhantomJS to PATH

Add the /bin folder of PhantomJS into your $PATH OS to make it easier when calling PhantomJS main binary. This allows us to call the binary anywhere without specifying the full path.

We will be using the Mac OS X version of PhantomJS throughout this book for running code examples. Don't worry if you are working on a different platform; the instructions are the same on all platforms.

Tip

Quick PhantomJS install on Mac OS X

As an alternative to downloading the precompiled binary, we can install PhantomJS using brew:

brew update && brew install phantomjs

For more information about brew, visit http://brew.sh/.

 

Building PhantomJS from source


You may also want to build your own binaries by compiling PhantomJS from source. Sources are hosted in the Github server at https://github.com/ariya/phantomjs.

Before you start downloading sources, you will need these tools installed on your workspace:

OS

Required development tools

Windows

Visual Studio 2010 or 2008 (Express edition)

git

Mac OS X

Xcode

git

Ubuntu/RHEL/CentOS Linux

gcc

gcc-c++

make

git

openssl-devel

freetype-devel

fontconfig-devel

The PhantomJS team is always trying to find the optimal way to build the sources, and the build instructions are frequently modified. To build PhantomJS properly, you must follow the steps found here: http://phantomjs.org/build.html.

If you are not planning to hack into PhantomJS code and develop new features, then it is best to download the pre-packaged binaries.

 

Working with PhantomJS


Now, let's see how PhantomJS's magic works. It is a command-line-based application, so we need to execute it in an OS terminal or console. The PhantomJS package contains a series of files and comes with one main executable file, which is named phantomjs.

Open your terminal and then navigate to your PhantomJS bin folder. In the prompt, execute phantomjs without any arguments.

Tip

PhantomJS Windows build

In Windows build, PhantomJS executable can be found in the root folder with the filename phantomjs.exe.

Running PhantomJS without any arguments will give you an interactive prompt that is similar to the JavaScript debug console you could find in any modern browser. In this interactive prompt, we can execute JavaScript code line by line. This functionality is very useful for debugging or testing code before you actually build your script.

Say "Hello Ghost!" to PhantomJS using the interactive prompt. Using console.log will output any type of data to the output console of a JavaScript interpreter.

phantomjs> console.log("Hello Ghost!")
Hello Ghost!
undefined
phantomjs>

See? It is simple. Just like coding any JavaScript. But wait. What is that undefined message just after the Hello Ghost! message? That is not an error. It is just how the interactive mode behaves. Each call is expected to return data just like any ordinary function call and it also automatically outputs the data value to the output stream.

Since the console.log command does not return any value, the message is undefined. If we issue an assignment to a variable command, the following output will be displayed:

phantomjs> name = "Tara"
{}
phantomjs>

The assignment to a variable will take place and the result of the operation will be displayed. Because it is in the form of a string literal, the undefined message will not be displayed. The interactive mode is similar to a long-running script; any variable or function you define will be loaded into the memory buffer and can be accessed anytime during the session. So, based on our preceding example, the name variable can also be displayed by referencing it.

phantomjs> name = "Tara"
"Tara"
phantomjs> name
"Tara"
phantomjs> name + " and Cecil"
"Tara and Cecil"
phantomjs>

We can even use the variable with another operation as seen in the preceding lines of code. However, any operation's result that is not assigned to a variable will be available only during the execution of the line. The operation that concatenates the name variable with another string literal will be performed, and the resulting string will be displayed in the console but will not be kept in memory.

Objects can also be accessed within the interactive mode, and one of the most commonly used objects is phantom. Try typing phantom in the prompt and you will get the following output:

phantomjs> phantom
{
   "clearCookies": "[Function]",
   "deleteCookie": "[Function]",
   "addCookie": "[Function]",
   "injectJs": "[Function]",
   "debugExit": "[Function]",
   "exit": "[Function]",
   "cookies": [],
   "cookiesEnabled": true,
   "version": {
      "major": 1,
      "minor": 7,
      "patch": 0
   },
   "scriptName": "",
   "outputEncoding": "UTF-8",
   "libraryPath": "/Users/Aries/phantomjs/bin",
   "defaultPageSettings": {
      "XSSAuditingEnabled": false,
      "javascriptCanCloseWindows": true,
      "javascriptCanOpenWindows": true,
      "javascriptEnabled": true,
      "loadImages": true,
      "localToRemoteUrlAccessEnabled": false,
      "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X)AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.7.0Safari/534.34",
      "webSecurityEnabled": true
   },
   "args": []
}
phantomjs>

PhantomJS displays the content of the object when used in the interactive prompt, and even its own phantom object can be referenced. You may also observe that the object is displayed in the form of JSON and details every attribute of the object except for the function definition. Using this approach, we can also examine each and every object, and we will be able to know what the exposed attributes and available functions are.

Let's try using one of the most important functions available in the phantom object: the exit() function. This function will enable us to quit PhantomJS and return to the caller or to the underlying operating system.

phantomjs> phantom.exit()
$

This function signals the application to exit with a return code of zero or normal and without errors. Passing a numeric value as an argument of the exit() function denotes the error code to be passed back to the caller. This is helpful when trying to write scripts that need to verify if the execution was successful or if an error occurred and what type of error it was.

If we trap the error code in a shell script, it will look as follows:

#!/bin/bash
bin/phantomjs
OUT=$?
if [ $OUT -eq 0 ];then
   echo "Done."
else
   echo "Ooops! Failed.!"
fi

In the preceding lines of code, right after calling phantomjs, we capture the error code coming from the application using the $? function. We assign that to an OUT variable and then perform a test on it in the succeeding lines. If the error is equal to zero, then we display Done; otherwise, we say that the call failed.

$ ./trapme.sh 
phantomjs> phantom.exit(0)
undefined
Done.
$ ./trapme.sh 
phantomjs> phantom.exit(1)
undefined
Ooops! Failed.!
$

Use the interactive mode to experiment with the PhantomJS API.

Before we begin creating PhantomJS scripts, we first need to make a quick roundup of the PhantomJS JavaScript API.

Tip

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

 

PhantomJS JavaScript API


PhantomJS runs JavaScript and comes with a JavaScript API to make your life easy. It extends the standard JavaScript API and adds richer layers of capabilities, such as allowing us access to the underlying file system, ease of access and manipulation of DOM objects, system and environment variable gathering. It even gives us the ability to inject custom scripts into the web page.

But, be warned. PhantomJS is a very active community, and every now and then, changes are being introduced. New APIs and objects are being added, but of course, there are few items that are being changed, and ultimately some of them become deprecated or are totally removed. The PhantomJS website has a full list of all the functions and syntax, and has proper tagging for deprecated functions. We should visit it regularly to check for upcoming updates so that we can adjust appropriately in our codes.

The Module API

While writing custom objects and API sets, you may want to create custom modules that will make your life easier. You can do that in PhantomJS using the Module API. It allows you to create your own modules and import it anywhere in your implementation. The built-in modules are webpage, system, fs (File System), and webserver.

The WebPage API

PhantomJS is a headless browser. Accessing and manipulating web documents are its core functionalities, and that's what the WebPage API is used for. The WebPage API allows us to access, control, and manipulate web documents. It provides a rich interface to easily reference and extract page details including document content. It enables capturing of events, such as page loading, when an error occurs within the page, or when navigating to another page is requested, and so on. It is also capable of capturing pages and saving them as images. And more importantly, it allows you to manipulate documents on the fly and traverse DOM as you do with any web page. This is enormously valuable for writing automated user interface tests—for example, you can force click events or post forms, and capture the results—as well as standard web scraping of public URLs.

The System API

The System module provides system-level functionalities ranging from OS information, environment variables, command-line arguments, and process-related properties. The System module is very useful as you engage more in developing applications with PhantomJS.

The FileSystem API

Accessing files, writing to text files, or just reading a custom configuration file—these are tasks that can be done with PhantomJS. FileSystem provides a standard API to perform file I/O. You can read, write, and delete files; you can even list folder files.

FileSystem has 31 functions to manage and manipulate files within PhantomJS. We have been using this set extensively, and it helps solve several problems without fail. Writing JSON data to a file is very basic, but you will find it much easier using the FileSystem API.

The WebServer API

Perhaps you have a grand idea, but it requires you to process a web request and execute a PhantomJS script on it. PhantomJS can do that; you can embed your own web server implementation using the WebServer API within your PhantomJS application. This feature is marked as experimental, but it does work, and with roughly five lines of code you can have your own web server running.

This module is based on the open source Mongoose web server library that supports multiple platforms, authorization, web sockets, URL rewrite, and even "resumeable" downloads. For more information about Mongoose, visit http://code.google.com/p/mongoose/.

 

The phantom object


The phantom object is your reference to PhantomJS within your scripts that allows you to access certain properties and provides functionality that affects the entire script (such as quitting the application as previously mentioned.) The phantom object can be directly referenced anywhere in the script and does not need to be explicitly imported. You may also access it as a child of the global window object.

The phantom object allows access to relevant data, such as cookies and library paths. If we want to import or inject third-party JavaScript libraries, such as jQuery, we can do that using the phantom object. We can also create a "catch all" event handler for errors using the onError event of the phantom object.

PhantomJS not only allows us to harness the power of JavaScript but also gives us a very useful API. Each day, more and more contributors are enhancing this API, giving us more options and easier ways to solve real problems. We will learn more about these APIs as we continue our journey learning about PhantomJS.

 

The command-line arguments


There are a few command-line arguments that we need to understand before plunging into writing PhantomJS scripts. The syntax of the PhantomJS argument is:

phantomjs [switches] [options] [script] [argument [argument [...]]]

All of the arguments are optional. Using the command without arguments will bring up the interactive mode.

The script argument

The script argument is the name of a script file. It can be a relative or an absolute path and must follow the path convention of the host system OS.

phantomjs /scripts/chapter1.js

The script filename may or may not end with a .js extension. PhantomJS supports two types of scripting—JavaScript or CoffeeScript. We don't need to specify if the script is in JavaScript or CoffeeScript; PhantomJS will automatically detect it. We will be using JavaScript in our examples. If you want to learn more about CoffeeScript, you may visit the CoffeeScript website at http://www.coffeescript.org.

The debug option

The debug option enables the printing of additional warnings and the debug messages. This is very useful when debugging your script. It accepts either yes or no as the value of the option, and no is the default value.

phantomjs --debug=yes /scripts/chapter1.js

The cookie-file option

If you are working on pages that require persistent cookies, you need to enable this option. This option will accept a file path where cookies will be saved and read.

phantomjs --cookie-file=/scripts/cookies.txt /scripts/chapter1.js
 

Writing PhantomJS scripts


We know how to write JavaScript, and now we know that there are several PhantomJS JavaScript APIs and objects. We also have learned the basics of the PhantomJS command-line arguments. We are now ready to create our own scripts.

We will create a simple script to load a site and then display the title of the page when loaded successfully. Finally, we will exit. If the page fails to load, we will log some message to the console.

var page = require('webpage').create();
page.open("http://www.packtpub.com", function(status) {
   if ( status === "success" ) {
      console.log(page.title); 
   } else {
      console.log("Page failed to load."); 
   }
   phantom.exit(0);
});

The preceding PhantomJS script is very simple. First, we import the webpage module, create an instance of the webpage object, and assign it to a variable named page.

The page variable now holds an instance of the webpage module where an open function is available. Next, we instructed PhantomJS through the webpage instance to open and load the URL. The second parameter of the open function is a function callback definition that will be executed upon completion of the opening of the URL. Inside the definition, we check if the status is "success", and if it is, the page is loaded, and then we will display the title of the page. Then, we call the exit function to terminate the script. Let's save this code snippet as helloweb.js and execute it by passing the filename as our first argument to the phantomjs binary.

 

Summary


At this point, we have learned quite a few basics of PhantomJS, but this is just the tip of the coolness of the technology. We now know how to get started with PhantomJS, and create small scripts to play around with it. We won't stop here. We will now move on to the more interesting stuff that it can offer, such as manipulating web pages, simulating user events, and grabbing screenshots. So, let's do more PhantomJS coding in the succeeding chapters.

About the Author

  • Aries Beltran

    Aries Beltran is a software developer located in Manila, Philippines. He works as an architect and R&D developer for financial businesses using web and enterprise technologies. He is currently developing new tools to provide real-time insights. He is interested in playing around with cutting-edge HTML5 development and mobile visualization. When he isn't coding, he likes to take photos of his daughter, Tara, who is his favorite model.

    Browse publications by this author
Book Title
Unlock this book and the full library for only $5/m
Access now