PhantomJS Cookbook

To install PhantomJS, we will need access to the Internet and permission to install applications.

Perform the following steps to download and install PhantomJS:

The version of PhantomJS that we just installed should print out to the console.

In an effort to lower the barrier to entry and help drive adoption, the prebuilt binaries of PhantomJS are made available by community volunteers. This is, in part, an acknowledgment that building PhantomJS from the source code can be a complex and time-consuming task. To quote the build page on the PhantomJS site: "With 4 parallel compile jobs on a modern machine, the entire process takes roughly 30 minutes." It is easy to imagine that this might scare off many developers who just want to try it out.

These prebuilt binaries should therefore make it easy to drop PhantomJS onto any system and have it running in minutes. These binaries are intended to be fully independent applications, with no external library dependencies such as Qt or WebKit. On some Linux systems, however, a little extra work may be required to ensure that the libraries necessary for proper font rendering (for example, FreeType and Fontconfig) are in place, along with the basic font files.

In addition to the prebuilt binaries, Mac OS X users may also install PhantomJS using Homebrew. To do this, enter the following as the command line:

brew update && brew install phantomjs

Note that installing PhantomJS with Homebrew also means that we will be compiling it from source.

To run this recipe, we will need to have PhantomJS installed on our PATH. We will also need an open terminal window.

Perform the following steps to invoke and work in the PhantomJS REPL:

The PhantomJS REPL, also called interactive mode, was introduced to PhantomJS starting with Version 1.5. The REPL is the default mode for PhantomJS when the application is invoked without any arguments.

REPL stands for Read-Evaluate-Print Loop. The commands we enter at the prompt are read by the interpreter, which evaluates them and prints the results, before finally looping back to the prompt for us to continue. Many programming environments feature REPLs (for example, Node.js provides another popular JavaScript REPL), and the debugger consoles in tools such as the Chrome Developer Tools and Firebug would also qualify as REPLs. REPLs are useful for quickly trying out ideas in the runtime environment.

In our example, we enter the PhantomJS REPL by invoking phantomjs from the command line without any arguments. Once we are in the REPL, we can type in whatever commands we need to explore our ideas, hitting Enter after each command. Note that we must enter a full and syntactically valid expression or statement before hitting Enter; if we do not, PhantomJS will report an error (for example, Can't find variable: foo or Parse error).

The PhantomJS REPL also features auto-completion. Hitting the Tab key in the REPL will autoexpand our options. We can even hit Tab multiple times to cycle through our available options; for example, try typing p and then hit Tab to see what options the REPL presents.

Finally, when we are finished, we use phantom.exit() to leave the REPL; we can also use the Ctrl + C or Ctrl + D key commands to exit the REPL.

To run this recipe, we will need PhantomJS installed on our PATH. We will also need a script to run with PhantomJS; the script in this recipe is available in the downloadable code repository as recipe03.js under chapter01. If we run the provided example script, we must change to the root directory for the book's sample code.

Given the following script:

console.log('A console statement from PhantomJS on ' +
  new Date().toDateString() + '!');

phantom.exit();

Type the following at the command line:

phantomjs chapter01/recipe03.js

Our preceding example script performs the following actions:

As we learned in the Launching the PhantomJS REPL recipe, PhantomJS will enter the REPL when invoked without any arguments. However, the runtime environment will attempt to evaluate and execute the first unrecognized argument as though it were a JavaScript file, regardless of whether or not it ends in .js. Most of the time that we work with PhantomJS, we will interact with it using scripts such as these.

As long as PhantomJS can resolve the first unrecognized argument as a file and correctly parse its contents as syntactically valid JavaScript, it will attempt to execute the contents. However, what happens if those preconditions are not met?

If the argument cannot be resolved as a file on disk, or if the file has no contents, PhantomJS will print an error message to the console, for example:

phantomjs does-not-exist-or-empty
Can't open 'does-not-exist-or-empty'

If the argument exists but the file's contents cannot be parsed as a valid JavaScript, then PhantomJS will print an error message to the console and hang, for example:

phantomjs invalid.js
SyntaxError: Parse error

Recall that PhantomJS is a headless web browser, and it helps to think of it as a version of Chrome or Safari that has no window. Just as we interact with our normal web browser by entering URLs into the location bar, clicking the back button, or clicking links on the page, so we will need to interact with PhantomJS. However, as it has no window and no UI components, we must interact with it through its programmable API. The PhantomJS API is written in JavaScript, and scripts targeting the PhantomJS runtime are also written in JavaScript; the API is documented online at http://phantomjs.org/api/.

If you have been exposed to both PhantomJS and Node.js, you may be wondering about the differences between them, especially after witnessing demonstrations of their respective REPLs and script running abilities. When comparing the two, it is helpful to consider them using the phrase "based on" as your frame of reference. Node.js is based on Google Chrome's V8 JavaScript engine; PhantomJS is based on the WebKit layout engine. Node.js is a JavaScript runtime; PhantomJS has a JavaScript runtime. Where Node.js is an excellent platform for building JavaScript-based server applications, it does not have any native HTML rendering. This is the key differentiator when comparing it to PhantomJS. The mission of PhantomJS is not to provide a platform for building JavaScript applications, but instead to provide a fast and standards-compliant headless browser.

To run this recipe, we will need a script to run with PhantomJS; the script in this recipe is available in the downloadable code repository as recipe04.js under chapter01. If we run the provided example script, we must change to the root directory for the book's sample code. Lastly, we will need the arguments we wish to pass into the script.

Given the following script:

if (phantom.args.length === 0) {
  console.log('No arguments were passed in.');
} else {
  phantom.args.forEach(function(arg, index) {
    console.log('[' + index + '] ' + arg);
  });
}

phantom.exit();

Enter the following command at the command line:

phantomjs chapter01/recipe04.js foo bar "boo baa"

We will see the following results printed in the terminal:

[0] foo
[1] bar
[2] boo baa

Our preceding example script performs the following actions:

As we discussed in the Running a PhantomJS script recipe, PhantomJS will attempt to evaluate and execute the first unrecognized argument as though it were a valid JavaScript file. But what does PhantomJS do with all of the arguments after that?

The answer is that they are collected into the phantom.args array as string values. Each argument after the script name goes into this array. Note that phantom.args does not include the script name itself. Instead, PhantomJS records that in the read-only phantom.scriptName property.

It is worth noting that both phantom.args and phantom.scriptName are both marked as deprecated in the API documentation. As such, usage of both of these properties is discouraged. Although using them for quick one-off or exploratory scripts is fine, neither of these properties should go into any library that we intend to maintain or distribute.

Wherever possible, we should use the system.args array (from the system module) instead of phantom.args and phantom.scriptName.

To run this recipe, we will need a script to run with PhantomJS that accesses a site where cookies are read or written. We will need a filesystem path to specify it as the command-line argument, making sure that we have write permissions to that path.

The script in this recipe is available in the downloadable code repository as recipe05.js under chapter01. If we run the provided example script, we must change to the root directory for the book's sample code.

Lastly, the script in this recipe runs against the demo site that is included with the cookbook's sample code repository. To run that demo site, we must have Node.js installed. In a separate terminal, change to the phantomjs-sandbox directory (in the sample code's directory) and start the app with the following command:

node app.js

npm install

Given the following script:

var webpage = require('webpage').create();

webpage.open('http://localhost:3000/cookie-demo', function(status) {
    if (status === 'success') {
      phantom.cookies.forEach(function(cookie, i) {
        for (var key in cookie) {
          console.log('[cookie:' + i + '] ' + key + ' = ' +
            cookie[key]);
        }
      });
    
      phantom.exit();
    } else {
      console.error('Could not open the page! (Is it running?)');
      phantom.exit(1);
    }
  });

Enter the following command at the command line:

phantomjs --cookies-file=cookie-jar.txt chapter01/recipe05.js

The script will print out the properties for each cookie in the response, as follows:

[cookie:0] domain = localhost
[cookie:0] expires = Sat, 07 Dec 2013 02:05:06 GMT
[cookie:0] expiry = 1386381906
[cookie:0] httponly = false
[cookie:0] name = dave
[cookie:0] path = /cookie-demo
[cookie:0] secure = false
[cookie:0] value = oatmeal-raisin
[cookie:1] domain = localhost
[cookie:1] expires = Sat, 07 Dec 2013 02:04:22 GMT
[cookie:1] expiry = 1386381862
[cookie:1] httponly = false
[cookie:1] name = rob
[cookie:1] path = /cookie-demo
[cookie:1] secure = false
[cookie:1] value = chocolate-chip

We can then open cookie-jar.txt in a text editor and examine its contents. The cookie jar file should look something like the following:

[General]
cookies="@Variant(\0\0\0\x7f\0\0\0\x16QList<QNetworkCookie>\0\0\0\0\x1\0\0\0\x2\0\0\0_dave=oatmeal-raisin; expires= Sat, 07 Dec 2013 02:05:06 GMT; domain=localhost; path=/cookie-demo\0\0\0^rob=chocolate-chip; expires= Sat, 07 Dec 2013 02:04:22 GMT; domain=localhost; path=/cookie-demo)"

Our preceding example script performs the following actions:

When we start PhantomJS with the cookies-file argument, we are telling the runtime to read and write cookies from a specific location on the filesystem. What this allows us to do is to use cookies in PhantomJS like we would with any other browser. In other words, an HTTP response or client-side script can set cookies, and when we run our PhantomJS script against that URL again, we can trust that the cookies are still there in the file.

Notice that the cookie jar file itself is essentially a plain text file. The actual file extension does not matter; we used .txt in our example, but it could just as easily be .cookies or even no extension at all. When persisting the cookies, PhantomJS writes them to this file. If we examine the file, then we see that it is a serialized, text-based version of the QNetworkCookie class that PhantomJS uses behind the scenes. Although the on-disk version is not necessarily easy to read, we can easily make a copy and parse it or transform it into its constituent cookies. This can be useful for examining their contents after a script has completed (for example, to ensure that the expected values are being written to disk).

Additionally, with the cookies written to disk, they are available for future PhantomJS script runs against URLs that expect the same cookies. For example, this can be useful when running scripts against sites that require authentication where those authentication tokens are passed around as cookies.

To run this recipe, we will need a script to run with PhantomJS that accesses a website with cacheable assets. Optionally, we will also need a sense of how large we wish to set the on-disk cache (in kilobytes).

The script in this recipe is available in the downloadable code repository as recipe06.js under chapter01. If we run the provided example script, we must change to the root directory for the book's sample code.

Lastly, the script in this recipe runs against the demo site that is included with the cookbook's sample code repository. To run that demo site, we must have Node.js installed. In a separate terminal, change into the phantomjs-sandbox directory (in the sample code's directory) and start the app with the following command:

node app.js

Given the following script:

var page  = require('webpage').create(),
    count = 0,
    until = 2;

page.onResourceReceived = function(res) {
  if (res.stage === 'end') {
    console.log(JSON.stringify(res, undefined, 2));
  }
};

page.onLoadStarted = function() {
  count += 1;
  console.log('Run ' + count + ' of ' + until + '.');
};

page.onLoadFinished = function(status) {
  if (status === 'success') {
    if (count < until) {
      console.log('Go again.\n');
      page.reload();
    } else {
      console.log('All done.');
      phantom.exit();
    }
  } else {
    console.error('Could not open page! (Is it running?)');
    phantom.exit(1);
  }
};

page.open('http://localhost:3000/cache-demo');

Enter the following command at the command line:

phantomjs --disk-cache=true --max-disk-cache-size=4000 chapter01/recipe06.js

The script will print out details about each resource in the response as JSON.

Our preceding example script performs the following actions:

Even though the disk cache is off by default, PhantomJS still performs some in-memory caching. This detail becomes important in later explorations, as it produces some otherwise difficult to explain results. For example, in our preceding sample script, we used webpage.reload for our second request of the URL, and in that second request, we saw all of the images re-requested. However, if we had used a second call to webpage.open (instead of webpage.reload), then the onResourceReceived callback would have shown a second request to the URL but none of the images would have been re-requested. (As an interesting aside, we would also see that behavior if we set the disk-cache argument to false; the in-memory cache cannot be disabled.)

Another interesting observation is that PhantomJS always reports an HTTP response status of 200 Ok for every successfully retrieved asset. If we look at the Node.js console output for the demo app while our sample script runs, we can see the discrepancy. Again, when our sample script runs, we can see that an HTTP status code of 200 is reported by PhantomJS for each of the images during both the first and second request/response cycles. However, the output from the Node.js app looks something like this:

GET /cache-demo 200 1ms - 573b
GET /images/583519989_1116956980_b.jpg 200 4ms - 264.64kb
GET /images/152824439_ffcc1b2aa4_b.jpg 200 8ms - 615.21kb
GET /images/357292530_f225d7e306_b.jpg 200 6ms - 497.98kb
GET /images/391560246_f2ac936f6d_b.jpg 200 5ms - 446.68kb
GET /images/872027465_2519a358b9_b.jpg 200 5ms - 766.94kb
GET /cache-demo 200 1ms - 573b
GET /images/152824439_ffcc1b2aa4_b.jpg 304 3ms
GET /images/357292530_f225d7e306_b.jpg 304 3ms
GET /images/391560246_f2ac936f6d_b.jpg 304 2ms
GET /images/583519989_1116956980_b.jpg 304 3ms
GET /images/872027465_2519a358b9_b.jpg 304 3ms

We can see that the server responds with 304 Not Modified for each of the image assets. This is exactly what we would expect for a second request to the same URL when the assets are served with Cache-Control headers that specify a max-age, and for assets that are also cached to disk.

To run this recipe, we will need a JSON-formatted configuration file with our PhantomJS command-line options.

The script in this recipe is available in the downloadable code repository as recipe07.js under chapter01. If we run the provided example script, we must change to the root directory for the book's sample code. An example configuration file is also in this directory as recipe07-config.json.

Lastly, the script in this recipe runs against the demo site that is included with the cookbook's sample code repository. To run that demo site, we must have Node.js installed. In a separate terminal, change to the phantomjs-sandbox directory (in the sample code's directory) and start the app with the following command:

node app.js

Select our command-line configuration options (changing hyphenated property names into their camel-cased equivalents) and apply our values. Save these configuration settings to a JSON-formatted document. For example, the contents of recipe07-config.json under chapter01:

{
  "cookiesFile"     : "cookie-jar.txt",
  "ignoreSslErrors" : true
}

Given the script from the Running PhantomJS with cookies recipe earlier in this chapter, enter the following at the command line:

phantomjs --config=chapter01/recipe07-config.json chapter01/recipe07.js

The configuration file is a JSON document where we can take our preferred command-line arguments and store them on disk. The keys in the JSON object have a one-to-one correspondence with the command-line arguments themselves – the hyphenated command-line argument names are converted to their camel-cased versions (for example, cookies-file becomes cookiesFile). The values in the JSON object follow easy conversion rules based on the most applicable JavaScript primitives: strings are strings, numbers are numbers, and true/false or yes/no become the corresponding true or false Boolean literals. Creating our own JSON-formatted configuration file requires only two things: a text editor and the knowledge of which command-line arguments we wish to capture in it.

The example script in this recipe (recipe07.js under chapter01) is identical to the one that we used for our demonstration in the Running PhantomJS with cookies recipe; we are reusing it here for convenience. For a more thorough explanation of what it is doing, see the How it works… section under that recipe.

When launching PhantomJS with the config command-line argument, the PhantomJS runtime interprets the argument's value as a path on the filesystem and attempts to load and evaluate that file as a JSON document. If the file cannot be parsed as a JSON document, then PhantomJS prints a warning and ignores it. If the file is correctly parsed, then PhantomJS configures itself as if the arguments in the JSON document had been passed as normal command-line arguments.

This raises an interesting question: given equivalent arguments, which one takes precedence? The one specified in the JSON configuration file? Or the one specified on the command line? The answer is that it depends which one comes last. In other words, given recipe07-config.json, we can run:

phantomjs --cookies-file=jar-of-cookies.txt --config=chapter01/recipe07-config.json chapter01/recipe07.js

That creates cookie-jar.txt, as specified in recipe07-config.json. While the following command creates jar-of-cookies.txt, as specified on the command line:

phantomjs --config=chapter01/recipe07-config.json --cookies-file=jar-of-cookies.txt chapter01/recipe07.js

Saving a PhantomJS configuration to a JSON document can help us in a couple of ways. First, by putting it into a file, we can put it under version control and track the changes to that configuration over time. Also, by putting the configuration into a file, it can more easily be shared across teams or jobs in continuous integration.

To run this recipe, we will need the following:

The script in this recipe is available in the downloadable code repository as recipe08.js under chapter01. If we run the provided example script, we must change to the root directory of the book's sample code.

The script in this recipe runs against the demo site that is included with the cookbook's sample code repository. To run that demo site, we must have Node.js installed. In a separate terminal, change to the phantomjs-sandbox directory and start the app with the following command:

node app.js

Given the following script:

var page = require('webpage').create();

page.onResourceReceived = function(res) {
  if (res.stage === 'end') {
    console.log(JSON.stringify(res, undefined, 2));
  }
};

page.open('http://localhost:3000/cache-demo', function(status) {
  if (status === 'success') {
    console.log('All done.');
    phantom.exit();
  } else {
    console.error('Could not open page! (Is it running?)');
    phantom.exit(1);
  }
});

Enter the following at the command line:

phantomjs --remote-debugger-port=9000 --remote-debugger-autorun=true chapter01/recipe08.js

Note that with the remote-debugger-autorun argument set to true, the script will run immediately as it normally would, but it will also ignore calls to phantom.exit and suspend execution, printing out the following message:

Phantom::exit() called but not quitting in debug mode.

Now we can open our other browser (for example, Chrome) and enter our IP address and the port that we specified with remote-debugger-port. For example, if our computer's IP address is 10.0.1.8, we would enter http://10.0.1.8:9000/ into the location bar. Then, we should see something like the following screenshot:

The viewport will contain the PhantomJS browsing session's history as a list. As we are interested in accessing the debugger tools, we will click on the link that reads about:blank. This will take us to /webkit/inspector/inspector.html, and it should look something like the following screenshot:

If we have worked in the Chrome or Safari developer tools before, the toolbar should be familiar. While debugging PhantomJS scripts, we will be particularly interested in the Scripts and Console tabs.

Once we have the debugger open, click on the Scripts tab. In the Scripts tab, click on the drop-down menu (in the top toolbar, just below the tabs) and select about:blank. This will show us our script as seen in the following screenshot. Click on any line number in the left-side gutter to set a breakpoint.

With our breakpoint set, click on the Console tab to toggle into the console. Since we used the remote-debugger-autorun argument, we will see our console.log and other such statements printed to the console from our first (automatic) run. Note the blue prompt at the bottom of the console as seen in the following screenshot; we can enter new expressions to be evaluated here at this prompt. To run our PhantomJS script again, we enter __run().

Entering __run() in the console will execute the script again. The script execution will pause on any breakpoints that we set and we will automatically be brought into the Scripts tab. In the Scripts tab, we can inspect our call stack, inspect local variables and objects at runtime, manipulate the runtime environment through the console, and more.

When we are done debugging our script, we can simply close the browser and then use Ctrl + C to quit the PhantomJS process in the terminal.

Our preceding example script is a simple one. We proceed in the following manner:

Effective debugging is an essential skill for every developer, and it is fantastic that PhantomJS has the WebKit remote debugging built-in as a first-class tool. While the debugger itself may be overkill for simple situations, sometimes console.log just isn't a powerful enough (or fast enough) tool. In those cases, it is comforting to know that you have these debug tools at your disposal.

One important thing to note about using the remote debugger with PhantomJS is that we will need to be aware of what context we are attempting to debug. Are we debugging the PhantomJS script itself? Or a script on the page that the PhantomJS script is accessing? Or some interaction between them? In the simple case (as previously demonstrated), the remote debug mode makes it almost trivial to inspect our PhantomJS script's execution at runtime. However, it does take some extra work if we need to also debug a script on the page that PhantomJS is accessing. In those cases, we may find it useful to use the remote-debugger-autorun argument; this will pre-populate the debugger's landing page with links to the inspector for the PhantomJS script's context and also the accessed web page's context. We can open these links each in a new tab, giving a separate debugger session for each context we need to work in.