PhantomJS is a new solution that provides headless testing of web applications. It is also a tool for dynamically capturing and rendering pages as images. It allows you to programmatically manipulate page content to control and change it to different forms. It can scrape websites and save important information to files. It will also provide you network-level information of your page and site resources. These are just a few of the functions that PhantomJS can do for us. It provides a fresh and a whole new way for web designers, testers, and developers to perform and create browser-based solutions.
PhantomJS uses QtWebKit as its core browser capability and uses the WebKit JavaScript engine for script interpretation and execution. Anything and everything that you can do in a WebKit-based browser (such as Chrome, Safari, and Opera browser) you can do with PhantomJS. It's more than just a browser because it supports web standards, such as CSS selector, DOM manipulation, JSON, HTML5 Canvas, and SVG; moreover, you can do some cool stuff such as performing file system I/O, accessing system environment variables, or even instantiating your own implementation of a web server daemon.
Downloading PhantomJS
Before we go through the features of PhantomJS, first we need to get our copy of the PhantomJS binaries. Typically, PhantomJS provides downloadable releases of binaries that are precompiled and packaged. You can choose from Linux, Mac OS X, and Windows precompiled packages. To download a copy, go to http://www.phantomjs.org/download.html.
Download your binaries based on your preference of operating system. After downloading, extract the binaries to any folder you desire. That's it! Your PhantomJS binary is ready to be used.
We will be using the Mac OS X version of PhantomJS throughout this book for running code examples. Don't worry if you are working on a different platform; the instructions are the same on all platforms.
Tip
Quick PhantomJS install on Mac OS X
As an alternative to downloading the precompiled binary, we can install PhantomJS using brew:
brew update && brew install phantomjs
For more information about brew, visit http://brew.sh/.
You may also want to build your own binaries by compiling PhantomJS from source. Sources are hosted in the Github server at https://github.com/ariya/phantomjs.
Before you start downloading sources, you will need these tools installed on your workspace:
OS |
Required development tools |
---|---|
Windows |
Visual Studio 2010 or 2008 (Express edition) git |
Mac OS X |
Xcode git |
Ubuntu/RHEL/CentOS Linux |
gcc gcc-c++ make git openssl-devel freetype-devel fontconfig-devel |
The PhantomJS team is always trying to find the optimal way to build the sources, and the build instructions are frequently modified. To build PhantomJS properly, you must follow the steps found here: http://phantomjs.org/build.html.
If you are not planning to hack into PhantomJS code and develop new features, then it is best to download the pre-packaged binaries.
Now, let's see how PhantomJS's magic works. It is a command-line-based application, so we need to execute it in an OS terminal or console. The PhantomJS package contains a series of files and comes with one main executable file, which is named phantomjs
.
Open your terminal and then navigate to your PhantomJS bin
folder. In the prompt, execute phantomjs
without any arguments.
Running PhantomJS without any arguments will give you an interactive prompt that is similar to the JavaScript debug console you could find in any modern browser. In this interactive prompt, we can execute JavaScript code line by line. This functionality is very useful for debugging or testing code before you actually build your script.
Say "Hello Ghost!" to PhantomJS using the interactive prompt. Using console.log
will output any type of data to the output console of a JavaScript interpreter.
phantomjs> console.log("Hello Ghost!") Hello Ghost! undefined phantomjs>
See? It is simple. Just like coding any JavaScript. But wait. What is that undefined message just after the Hello Ghost! message? That is not an error. It is just how the interactive mode behaves. Each call is expected to return data just like any ordinary function call and it also automatically outputs the data value to the output stream.
Since the console.log
command does not return any value, the message is undefined. If we issue an assignment to a variable command, the following output will be displayed:
phantomjs> name = "Tara" {} phantomjs>
The assignment to a variable will take place and the result of the operation will be displayed. Because it is in the form of a string literal, the undefined message will not be displayed. The interactive mode is similar to a long-running script; any variable or function you define will be loaded into the memory buffer and can be accessed anytime during the session. So, based on our preceding example, the name
variable can also be displayed by referencing it.
phantomjs> name = "Tara" "Tara" phantomjs> name "Tara" phantomjs> name + " and Cecil" "Tara and Cecil" phantomjs>
We can even use the variable with another operation as seen in the preceding lines of code. However, any operation's result that is not assigned to a variable will be available only during the execution of the line. The operation that concatenates the name variable with another string literal will be performed, and the resulting string will be displayed in the console but will not be kept in memory.
Objects can also be accessed within the interactive mode, and one of the most commonly used objects is phantom
. Try typing phantom
in the prompt and you will get the following output:
phantomjs> phantom { "clearCookies": "[Function]", "deleteCookie": "[Function]", "addCookie": "[Function]", "injectJs": "[Function]", "debugExit": "[Function]", "exit": "[Function]", "cookies": [], "cookiesEnabled": true, "version": { "major": 1, "minor": 7, "patch": 0 }, "scriptName": "", "outputEncoding": "UTF-8", "libraryPath": "/Users/Aries/phantomjs/bin", "defaultPageSettings": { "XSSAuditingEnabled": false, "javascriptCanCloseWindows": true, "javascriptCanOpenWindows": true, "javascriptEnabled": true, "loadImages": true, "localToRemoteUrlAccessEnabled": false, "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X)AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.7.0Safari/534.34", "webSecurityEnabled": true }, "args": [] } phantomjs>
PhantomJS displays the content of the object when used in the interactive prompt, and even its own phantom
object can be referenced. You may also observe that the object is displayed in the form of JSON and details every attribute of the object except for the function definition. Using this approach, we can also examine each and every object, and we will be able to know what the exposed attributes and available functions are.
Let's try using one of the most important functions available in the phantom
object: the exit()
function. This function will enable us to quit PhantomJS and return to the caller or to the underlying operating system.
phantomjs> phantom.exit() $
This function signals the application to exit with a return code of zero or normal and without errors. Passing a numeric value as an argument of the exit()
function denotes the error code to be passed back to the caller. This is helpful when trying to write scripts that need to verify if the execution was successful or if an error occurred and what type of error it was.
If we trap the error code in a shell script, it will look as follows:
#!/bin/bash bin/phantomjs OUT=$? if [ $OUT -eq 0 ];then echo "Done." else echo "Ooops! Failed.!" fi
In the preceding lines of code, right after calling phantomjs
, we capture the error code coming from the application using the $?
function. We assign that to an OUT
variable and then perform a test on it in the succeeding lines. If the error is equal to zero, then we display Done; otherwise, we say that the call failed.
$ ./trapme.sh phantomjs> phantom.exit(0) undefined Done. $ ./trapme.sh phantomjs> phantom.exit(1) undefined Ooops! Failed.! $
Use the interactive mode to experiment with the PhantomJS API.
Before we begin creating PhantomJS scripts, we first need to make a quick roundup of the PhantomJS JavaScript API.
Tip
Downloading the example code
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
PhantomJS runs JavaScript and comes with a JavaScript API to make your life easy. It extends the standard JavaScript API and adds richer layers of capabilities, such as allowing us access to the underlying file system, ease of access and manipulation of DOM objects, system and environment variable gathering. It even gives us the ability to inject custom scripts into the web page.
But, be warned. PhantomJS is a very active community, and every now and then, changes are being introduced. New APIs and objects are being added, but of course, there are few items that are being changed, and ultimately some of them become deprecated or are totally removed. The PhantomJS website has a full list of all the functions and syntax, and has proper tagging for deprecated functions. We should visit it regularly to check for upcoming updates so that we can adjust appropriately in our codes.
While writing custom objects and API sets, you may want to create custom modules that will make your life easier. You can do that in PhantomJS using the Module API. It allows you to create your own modules and import it anywhere in your implementation. The built-in modules are webpage
, system
, fs (File System)
, and webserver
.
PhantomJS is a headless browser. Accessing and manipulating web documents are its core functionalities, and that's what the WebPage API is used for. The WebPage API allows us to access, control, and manipulate web documents. It provides a rich interface to easily reference and extract page details including document content. It enables capturing of events, such as page loading, when an error occurs within the page, or when navigating to another page is requested, and so on. It is also capable of capturing pages and saving them as images. And more importantly, it allows you to manipulate documents on the fly and traverse DOM as you do with any web page. This is enormously valuable for writing automated user interface tests—for example, you can force click events or post forms, and capture the results—as well as standard web scraping of public URLs.
The System module provides system-level functionalities ranging from OS information, environment variables, command-line arguments, and process-related properties. The System module is very useful as you engage more in developing applications with PhantomJS.
Accessing files, writing to text files, or just reading a custom configuration file—these are tasks that can be done with PhantomJS. FileSystem provides a standard API to perform file I/O. You can read, write, and delete files; you can even list folder files.
FileSystem has 31 functions to manage and manipulate files within PhantomJS. We have been using this set extensively, and it helps solve several problems without fail. Writing JSON data to a file is very basic, but you will find it much easier using the FileSystem API.
Perhaps you have a grand idea, but it requires you to process a web request and execute a PhantomJS script on it. PhantomJS can do that; you can embed your own web server implementation using the WebServer API within your PhantomJS application. This feature is marked as experimental, but it does work, and with roughly five lines of code you can have your own web server running.
This module is based on the open source Mongoose web server library that supports multiple platforms, authorization, web sockets, URL rewrite, and even "resumeable" downloads. For more information about Mongoose, visit http://code.google.com/p/mongoose/.
The phantom
object is your reference to PhantomJS within your scripts that allows you to access certain properties and provides functionality that affects the entire script (such as quitting the application as previously mentioned.) The phantom
object can be directly referenced anywhere in the script and does not need to be explicitly imported. You may also access it as a child of the global window
object.
The phantom
object allows access to relevant data, such as cookies and library paths. If we want to import or inject third-party JavaScript libraries, such as jQuery, we can do that using the phantom
object. We can also create a "catch all" event handler for errors using the onError
event of the phantom
object.
PhantomJS not only allows us to harness the power of JavaScript but also gives us a very useful API. Each day, more and more contributors are enhancing this API, giving us more options and easier ways to solve real problems. We will learn more about these APIs as we continue our journey learning about PhantomJS.
There are a few command-line arguments that we need to understand before plunging into writing PhantomJS scripts. The syntax of the PhantomJS argument is:
phantomjs [switches] [options] [script] [argument [argument [...]]]
All of the arguments are optional. Using the command without arguments will bring up the interactive mode.
The script
argument is the name of a script file. It can be a relative or an absolute path and must follow the path convention of the host system OS.
phantomjs /scripts/chapter1.js
The script filename may or may not end with a .js
extension. PhantomJS supports two types of scripting—JavaScript or CoffeeScript. We don't need to specify if the script is in JavaScript or CoffeeScript; PhantomJS will automatically detect it. We will be using JavaScript in our examples. If you want to learn more about CoffeeScript, you may visit the CoffeeScript website at http://www.coffeescript.org.
We know how to write JavaScript, and now we know that there are several PhantomJS JavaScript APIs and objects. We also have learned the basics of the PhantomJS command-line arguments. We are now ready to create our own scripts.
We will create a simple script to load a site and then display the title of the page when loaded successfully. Finally, we will exit. If the page fails to load, we will log some message to the console.
var page = require('webpage').create(); page.open("http://www.packtpub.com", function(status) { if ( status === "success" ) { console.log(page.title); } else { console.log("Page failed to load."); } phantom.exit(0); });
The preceding PhantomJS script is very simple. First, we import the webpage
module, create an instance of the webpage
object, and assign it to a variable named page
.
The page
variable now holds an instance of the webpage
module where an open
function is available. Next, we instructed PhantomJS through the webpage
instance to open and load the URL. The second parameter of the open
function is a function callback definition that will be executed upon completion of the opening of the URL. Inside the definition, we check if the status is "success
", and if it is, the page is loaded, and then we will display the title of the page. Then, we call the exit
function to terminate the script. Let's save this code snippet as helloweb.js
and execute it by passing the filename as our first argument to the phantomjs
binary.
At this point, we have learned quite a few basics of PhantomJS, but this is just the tip of the coolness of the technology. We now know how to get started with PhantomJS, and create small scripts to play around with it. We won't stop here. We will now move on to the more interesting stuff that it can offer, such as manipulating web pages, simulating user events, and grabbing screenshots. So, let's do more PhantomJS coding in the succeeding chapters.