Before we can start building our application, we should set up a suitable development environment. This should as closely as possible mirror the final live server configuration, so that we can test our software locally before deploying it. The environment should also provide appropriate tools to support us during development.

In this article we will learn more about the elements of Zope's software stack and how they can be configured for development. We will also cover some supporting tools and technologies, such as Python eggs and Paste Script.

Prerequisites

Pre-built packages for Zope and Plone are available for many operating systems. These can be tempting, but as developers it is normally better to configure the environment ourselves, in order to fully understand and control it. During development, we need write access to the Python code and configuration files. We may also need to run different versions of Zope and Plone in parallel. We will assume that you have at least the following as part of your regular development environment:

• Python 2.4. We will assume that Python 2.4 gets invoked when you run python on the command line. Unfortunately, Zope, at version 2.10, does not yet support Python 2.5. Note that many Linux distributions have two packages for Python—one containing the binaries, and one containing header files (typically called python-dev or something similar). You will need both in order to compile Zope.
  
  
• PIL, the Python Imaging Library (http://www.pythonware.com/products/pil), should be installed for this Python interpreter.
  
  
• elementtree, an XML processing library, is required for Plone to start up. Most operating systems have packages for this. It can also be downloaded from http://effbot.org/zone/element-index.htm.
  
  
• A programmer's text editor. Preferably one with Python, XML/HTML and CSS syntax highlighting. You should set up your editor so that a tab/indent is output as four spaces. This makes Python development a lot more predictable.
  
  
• A shell. Most examples in this book will show a Bash interpreter shell, though we will cover Windows syntax when it differs significantly. Bear in mind that path separators on Windows are backslashes (), while other environments use forward slashes (/). Also, environment variables on Windows are referred to as %NAME%, while in most Unix shells, including Bash, variables are dereferenced with $NAME.
  
  
• A Subversion client. We will show the command line client syntax, but you can use a graphical client if you are more comfortable with that. Subversion can be obtained from http://subversion.tigris.org.
  
  
• A C compiler. You will need this to compile Zope. The venerable gcc is fine on UNIX-like systems. On Windows you probably want mingw32 (http://www.mingw.org). Alternatively, you can use a Zope Windows installer to get a binary Zope distribution.

Quick Start

Understanding your development environment is an important step in becoming a productive developer. If you need to get up and running quickly, however, and you have the prerequisites outlined above in order, here are the key steps.

We will assume you have Python 2.4 as your main Python interpreter. First download easy_install if you do not have it already, and use it to install ZopeSkel:

  $ wget http://peak.telecommunity.com/dist/ez_setup.py
  $ python ez_setup.py
  $ easy_install ZopeSkel

Note: If you do not have wget (e.g. because you are using Windows), you can just as easily download the ez_setup.py script using a web browser. When you run this script, it will install the easy_install binary to a directory that may not already be in your system PATH. Watch the ez_setup.py output to identify this directory. If it is not in your PATH, you should add it, allowing you to run easy_install as shown.

Then, use paster, which was installed as a dependency of ZopeSkel, to create a new buildout. This folder holds our source code and dependencies, including the Zope application server:

  $ paster create -t plone3_buildout myproject

You can accept the defaults for all the questions, except for the password, which you must enter. Then, build the environment like so:

  $ cd myproject
  $ python bootstrap.py
  $ ./bin/buildout

Note: If you are using Windows, be sure to read the README.txt file that is generated in the myproject directory before running the buildout command.

This last step may take some time, and you will need a live Internet connection.When it is complete, you can start Zope with:

  $ ./bin/instance fg

Go to http://localhost:8080/manage and you should see the Zope Management Interface. Use the drop-down box to add a Plone Site. If you call this mysite, it will beaccessible from http://localhost:8080/mysite.

Glossary

Let us now take a step back and consider our development environment in more detail.

The table below summarizes the various terms and technologies that you will encounter in this chapter. It pays to be familiar with these names, because you will find them again not only throughout this book, but also in other Plone documentation.

Understanding Eggs and Setuptools

Python eggs are not specific to Zope or Plone. However, since Zope has only recently become egg-aware, they are new to many developers.

Traditionally, almost all Zope add-on products, including Plone, have been distributed as Zope products. These are fairly easy to manage—you typically copy or symlink them into $INSTANCE_HOME/Products. Zope will scan this directory on startup, taking care of any product initialization and registration.

However, code inside products is nearly impossible to re-use outside Zope because Zope does magic things with the Products.* namespace. Further, the namespace quickly becomes crowded, which deters developers from breaking up functionality into smaller, more re-usable and better-isolated packages.

The Zope 3 philosophy is to be as close to "plain Python" as possible, and that means distributing code as such small packages. So long as its dependencies are in order, any package should be able to run in any environment. For example, the zope.interface package is used by the Twisted project, which is not otherwise dependent on Zope. This design goal has made it much easier to adopt Zope 3 packages in Zope2 and Plone.

Starting with Plone 3, the Plone community has also embraced "plain Python" packages and uses them wherever possible. A number of packages, such as plone.memoize and plone.portlets are generic enough to work without any dependencies on the rest of Plone. Others are more specific to Plone and live in the plone.app namespace, such as plone.app.layout and plone.app.portlets, thelatter containing Plone-centric extensions to the generic plone.portlets package.

All that is needed to use these packages is a sensible $PYTHONPATH. Thus, we can copy or link packages into $INSTANCE_HOME/lib/python/plone/memoize, lib/python/plone/portlets, lib/python/plone/app/portlets, and so forth for Zope to find them. This works, but it is pretty tedious when there are many packages, and it can become outright confusing when there are nested namespaces being used bymultiple packages.

Luckily, other Python programmers have solved these problems, first creating distutils, then its successor setuptools and with setuptools, Python eggs.

Note: For the setuptools documentation, see http://peak.telecommunity.com/DevCenter/setuptools.

Installing Eggs

When using setuptools, each project or package lives in a directory that has a toplevel setup.py file. This contains metadata about the package itself, and declares its current version as well as any dependencies. Dependencies can be specifieddown to particular versions (e.g. ">=0.2,<1.0" means "later than version 0.2 but before version 1.0"). When a package is installed, setuptools will attempt to fulfilldependencies by downloading and installing them if necessary.

If you have a setuptools-enabled package, you can use setup.py to install it globally, by running:

  $ python setup.py install

  $ python setup.py develop

New packages can be released as binary eggs for distribution, which are just ZIP files of the package with some additional metadata. You can build an egg from within apackage by running:

  $ python setup.py bdist_egg

The new egg will be placed in the dist sub-directory, which will be created if necessary.

You will be asked to specify or create a Cheese Shop account if this is the first time you run this command. A script called easy_install lets you search the Cheese Shop (or a similar index, if you specify a URL) for packages that it can download and install into the global Python environment. Dependencies will be included automatically. This is great for simple libraries and end-user applications, but less great when you are working on multiple Zope projects that may have different version requirements. This is why we tend to manage our eggs inside $INSTANCE_HOME or, as you will see in the next section, as part of a controlled buildout.

Note: A tool called workingenv.py can create a mini-environment where global commands are restricted to a particular directory. We will not cover workingenv in this book, but you can download and read more about it at http://cheeseshop.python.org/pypi/workingenv.py.

When eggs are activated (either explicitly, or implicitly by being unambiguouslyfound in the $PYTHONPATH), they can be discovered by other packages listening for plug-ins, using a mechanism called entry points (see http://peak.telecommunity.com/DevCenter/setuptools#dynamic-discovery-of-services-and-plugins).

Zope does not yet directly use entry points, so we will not be covering them inany detail here. However, entry points are a very powerful system, and there areproposals to let Zope's discovery of packages use entry points instead of scanningmagic directories.

With eggs, we therefore have the tools to manage multiple packages, from differentdevelopers and repositories, possibly across multiple versions. By using the packagemanagement tools that the rest of the Python community employs, we also make iteasier to re-use other libraries and share our own code with outside developers.

Automating the Build Process with zc.buildout

Creating a Zope instance and copying or linking packages into $INSTANCE_HOME/lib/python as is done in the manual process is not too difficult, but this approach has a few limitations.

• The process is manual and cumbersome to repeat across multiple environments.
• Multiple developers working on the same project may share the code in eggs and products by using a version control system such as Subversion. However, each developer would be responsible for setting up their development environment, and subtle differences may cause problems that are difficult to debug.
• Packages are installed manually, and so cannot benefit from setuptools' ability to manage dependencies and updates.
• Complex deployments that include other libraries, non-python code, or specific configurations will also need to be taken care of manually.

Luckily, there are tools to make deployment easier. zc.buildout is one such tool, written largely by Zope founder Jim Fulton at Zope Corporation. It makes heavyuse of eggs and setuptools and is very flexible in supporting a wide range of deployment scenarios.

Central to a buildout (i.e. what zc.buildout is managing for us) is a file called buildout.cfg. This specifies various options, including a list of parts, which will be executed when the buildout is run. Each part is associated with a recipe—a named egg, which will be called upon to parse the options provided, and perform a particular task, such as building a Zope instance or downloading Plone.

A project-specific buildout directory can be checked into a version control system and shared among developers. It can also be used to replicate a particular environment across different servers with a high degree of predictability.

By writing custom recipes, you can make zc.buildout do almost anything. Writing a recipe is not particularly hard, and there are plenty of examples and generic solutions available. However, we will not cover creating new recipes in this book, because all the recipes we need already exist.

Plone development fundamentals Customizing Plone Developing new functionality Real-world deployments For more information, please visit: http://www.packtpub.com/Professional-Plone-web-applications-CMS/book

Installing Paste Script and ZopeSkel

Paste Script is an extensible system with which other packages (or rather, eggs) can register new templates (using entry points). For Zope and Plone development, the ZopeSkel package provides several useful templates.

You can fetch ZopeSkel using easy_install. If you do not have easy_install, you must download it:

  $ wget http://peak.telecommunity.com/dist/ez_setup.py  $ python ez_setup.py

Keep an eye on where the script puts the easy_install executable. Depending on your setup, you may need to add this to your $PATH environment variable or reference it by an absolute directory. Adding the script's directory to your $PATH maybe a good idea, though, since the paster command and other egg-installed binaries will go in the same place.

Now, simply run:

  $ easy_install ZopeSkel

This will find ZopeSkel in the Cheese Shop, download it, and install it, including its dependencies.

Creating and Customizing the Buildout

  $ paster create -t plone3_buildout optilux

If you run this command, you will be asked a series of questions, including:

• The path to an existing appropriate installation of Zope 2. You can specify an absolute path here if you have installed Zope already and wish to share the same Zope installation across multiple buildouts. Leave the option blank to have buildout download and build Zope for you.
• The path to an existing directory containing all the Plone products. Again, this lets you share the same code across multiple buildouts and save some time. If you leave it blank, Plone will be downloaded for you.
• The Zope root user and password.
• The port that the Zope HTTP server will run on.
• Whether or not debug mode should be on by default. Note that even if it is off, you can enable debug mode by starting Zope with ./bin/instance fg instead of ./bin/instance start.
• Whether "verbose security" should be on by default. Verbose security is very useful for debugging security problems by offering more detailed log messages, but it is best turned off for production servers.

Buildout will create a new environment, containing a generated buildout.cfg file and some standard directories. To bootstrap the buildout to get the standard zc.buildout tools, run:

  $ cd optilux
  $ python bootstrap.py

This step is only needed once. We are now ready to build the system. Simply run:

  $ ./bin/buildout

This may take a long time, depending on the speed of your computer and internet connection. It will download and build Zope, download and install Plone, and configure the two. When it is finished, you can start Zope by running:

  $ ./bin/instance start

Stop it again with:

  $ ./bin/instance stop

The Buildout Configuration File

That is all that is needed to get started. To understand what is going on, though, let us take a look at the buildout.cfg file that was generated for us (slightly abbreviated):

    [buildout]
    parts =
        plone
        zope2
        productdistros
        instance
        zopepy
    find-links =
        http://dist.plone.org
        http://download.zope.org/distribution/
        http://effbot.org/downloads
    eggs =
        elementtree
    develop =
    [plone]
    recipe = plone.recipe.plone
    [zope2]
    recipe = plone.recipe.zope2install
    url = ${plone:zope2-url}
    [productdistros]
    recipe = plone.recipe.distros
    urls =
    nested-packages =
    version-suffix-packages =
    [instance]
    recipe = plone.recipe.zope2instance
    zope2-location = ${zope2:location}
    user = admin:admin
    http-address = 8080
    debug-mode = on
    verbose-security = on
    eggs =
        ${buildout:eggs}
        ${plone:eggs}
    zcml =
    products =
        ${buildout:directory}/products
        ${productdistros:location}
        ${plone:products}
    [zopepy]
    recipe = zc.recipe.egg
    eggs = ${instance:eggs}
    interpreter = zopepy
    extra-paths = ${zope2:location}/lib/python
    scripts = zopepy

Starting from the top, this file defi nes various things that will happen when our buildout is run:

• The main [buildout] section sets buildout-wide options. First, it lists several parts, which will be executed in order. These refer to sections later on in the file. Then, a number of eggs to install can be listed. By default, this includes elementtree, which is required by Plone. Even if you have elementtree in your system-wide Python installation already, listing it explicitly here will not hurt. The eggs are referenced again in the [instance] section, causing them to be downloaded from the Cheese Shop (or another source, as listed under find-links) and installed when the instance is built and needs to know about them. The empty develop option will let us manage development eggs later.
• The [plone] section will download Plone's products and eggs. It also exposes the URL to a "known good" version of Zope, which we will reference in the the [zope2] section. To peg Plone to a particular version, you can specify a version for the recipe, e.g. with recipe = plone.recipe.plone==3.0.1, which would make sure that we got the version of the release recipe corresponding to Plone 3.0.1, if and when it is released.
• The [zope2] section will download and build Zope 2 from the given URL, here supplied by the previously defined [plone] section. This will only be present if you did not specify an existing Zope installation above. It may be worth checking the URL to ensure you get the latest appropriate version of Zope.
• The [productdistros] section can be used to download and install archives of old-style products; it is referenced again under the products option of the [instance] section. We will add a few download URLs later in this chapter. The nested-packages and version-suffix-packages can be used to deal with archives that do not immediately extract to usable product directories. See the comments in the file for more information.
• The [instance] section is the most important one. This will create a newZope instance, and confi gure it with the appropriate options. The givenlist of eggs, referring back to the eggs in the main [buildout] section (although additional eggs can be added if necessary), will be activated and made available to Zope. Zope is also told where to look for products. The zcml section can be used to load ZCML slugs—special files that Zope 3-style packages can use to make themselves known to Zope. We will cover slugsin Chapter 9.
• The [zopepy] section sets up a custom Python interpreter, which will have available to it all the same eggs as the Zope instance. This is very useful for debugging and quick prototyping.

If for any reason buildout.cfg is changed, or if you wish to obtain recent updates to any eggs listed, you should run ./bin/buildout again. You can often save sometime by running buildout in offline and non-updating mode, where it will not check for updates to eggs and products online, by using:

  $ ./bin/buildout -No

To get a full explanation of the various options available, run:

  $ ./bin/buildout --help

The Buildout Directory

Let us now take a look at the directories zc.buildout creates for us.

• The bin directory contains the instance control script, zopepy, and the buildout command itself. ./bin/instance is equivalent to $INSTANCE_HOME/bin/zopectl.
• The eggs directory contains eggs that buildout has automatically downloaded. These are enumerated and referenced when needed, forexample by the instance control script. This is similar to linking packages into $INSTANCE_HOME/lib/python, but allows us to use the full power of setuptools and eggs.
• The develop-eggs directory contains egg-links to any development eggs specified in buildout.cfg. This will be empty for now, but we will make use of development eggs later in the book.
• A downloads directory will be created as necessary. Recipes such as plone.recipe.plone and plone.recipe.distros that fetch archives of products and packages will place their downloads in this directory. If an archive has already been downloaded, it will not be downloaded again.
• The products directory can be used for any old-style products that are being developed as part of the project. It is analogous to the aforementioned $INSTANCE_HOME/Products directory. Our buildout will also manage various products for us inside the parts directory.
• The src directory contains the sources of any custom eggs. This is empty now, but we will add packages here in later chapters.
• The var directory houses the Data.fs file and Zope's logs.
• The parts directory is zc.buildout's playground. This is where Zope 2 is downloaded and built, for example—the $SOFTWARE_HOME becoming parts/zope2/lib/python. The $INSTANCE_HOME is in parts/instance. You will not typically manage the Zope instance directly when using a buildout. You should not make changes to anything in the parts directory, as zc.buildout may stomp on your modifications when you re-run ./bin/buildout.

Avoiding Duplication between Buildouts

If as a developer you are managing several projects with different buildouts, you may want to share some source code between them. We have already seen the options that allow you to specify shared Zope 2 installations and Plone product directories. This will save considerable disk space, but bear in mind that old-styleZope products are not individually versioned, so any changes or upgrades will affect all instances where they are referenced.

It is also possible to share an eggs directory between different buildouts. Unlike products, eggs carry version information, and so it is possible to keep different versions installed simultaneously. For example, if one egg was referenced in buildout.cfg as my.package==0.9 and another buildout used my.package>=1.0, two versions could be downloaded and activated as appropriate.

The eggs directory to use is controlled by the eggs-directory option in the[buildout] section of buildout.cfg. Rather than having to add this to each project, however, we will put it in the defaults.cfg file, which zc.buildout examines to pickup additional options in addition to what is in the buildout.cfg file.

In your home directory, create a directory called .buildout. Within it, add another directory called eggs, and a file called default.cfg, containing:

    [buildout]
    eggs-directory = /home/username/.buildout/eggs

Similarly, you can specify a shared archive downloads directory, using the download-directory option. For example, you can create a directory called downloads next to the aforementioned eggs directory, and add the following to default.cfg:

    [buildout]
    eggs-directory = /home/username/.buildout/eggs
    download-directory = /home/username/.buildout/downloads

This file can be used for other defaults if necessary. In particular, if you need to use a specific Python interpreter instead of the default system-wide one, use a line like the following:

    executable = /path/to/python2.4

Plone development fundamentals Customizing Plone Developing new functionality Real-world deployments For more information, please visit: http://www.packtpub.com/Professional-Plone-web-applications-CMS/book

Additional Development Tools

During development, we typically rely on various tools to help us debug our code and inspect Plone and Zope. The oldest of these is the DocFinderTab, a Zope product that adds a Doc tab to every object in the ZMI. This tab lists the class and base classes for the object being inspected, including methods and their docstrings.

By using a regular expression in the Filter box at the top, you can look for specific methods.

If DocFinderTab is old-school, Clouseau is hip and happening. It is essentially an in-browser Python shell, using JavaScript and AJAX technologies. It supports two modes:

• A general interpreter prompt, which can be accessed from the Clouseau control panel under Site Setup.
  
• An object inspector, which can be invoked on any content object. It is available from the "document actions" shown at the bottom of most content views.

Here is an example of Clouseau inspecting the default Plone front page, changing the title and saving the results:

Please refer to the Clouseau help, under the Clouseau control panel, for more information.

Both DocFinderTab and Clouseau are distributed as old-style products in separate archives. To obtain and install them, edit buildout.cfg and modify the [productdistros] section like this:

    [productdistros]
    recipe = plone.recipe.distros
    urls =
        http://www.zope.org/Members/shh/DocFinderTab/1.0.1/
                    DocFinderTab-1.0.1.tar.gz
        http://plone.org/products/clouseau/releases/0.7.1/
                    Clouseau.0.7.1.zip

These URLs are correct at the time of writing. You may want to check for newerversions, especially of Clouseau, at http://plone.org/products/clouseau.

Now, we can re-run buildout and start Zope in debug mode:

  $ ./bin/buildout -N
  $ ./bin/instance fg

We should then be able to install Clouseau from the Plone control panel.DocFinderTab requires no explicit installation.

Learning to Help Yourself

During development, there will probably be times when you are stumped. Plone is fairly well-documented, but the documentation is certainly not perfect. The mailing lists and chat room are great resources if you need help, but it is also very importantto learn how to help yourself.

Use the Source, Luke!

Python's readability is both a blessing and a curse. A blessing, because it is normally possible to read the source code and find out what is going on. A curse, because this sometimes makes developers a little lax about documentation.

One of the first hurdles new developers should overcome is any undue respect for the Python files that make up Zope and Plone. There is (almost) nothing magical about them. In this chapter, we have seen where the source code lives: core Zope modules in $SOFTWARE_HOME, third-party products in the products folder, and eggs in lib/python or the eggs folder managed by buildout.

Get used to searching for code in these files using grep or equivalent graphical tools, opening them, and looking for specific classes and methods. Seeing what a piece of code does can often be faster than looking up documentation or examples. As time goes by, you will find that a few packages come up again and again, and finding code will be easier.

You can of course change these files as well. A backup is advisable, but if you think that temporarily raising an exception or printing a message from somewhere deep inside Zope helps you to solve a problem, go right ahead. It is probably a bad idea to make permanent changes this way, however, because those changes will be overwritten if you upgrade or re-install the particular component. In the next chapter, we will learn more about other ways of customizing code Zope and Plone. However, if you find a bug, please report it, and attach a patch if you can!

Note: Plone bugs can be reported at http://dev.plone.org/plone. Zope bugs are collected at http://zope.org/Collectors/Zope, and CMF bugs at http://zope.org/Collectors/CMF.

Here are some examples of where you may look for source code if you see an import or a description of a module in this book:

Become Familiar with the Debugger

It is also very important to be familiar with pdb, the Python debugger (see http://docs.python.org/lib/module-pdb.html). To insert a breakpoint in your code—or in some other code that you are trying to debug—add the following line and (re-)start Zope in the foreground in a terminal:

    import pdb; pdb.set_trace()

When the line is encountered, execution will stop, and the terminal will display:

  (pdb)

This is the interactive pdb prompt. Type help and press Enter to see available commands: the most important ones are pp—to print a variable or the result of an expression, n—to step to the next line, s—to step into a function call, l—to show a listing of the source code around the current execution point, and c—to stop debugging and continue execution. If you want to quickly test syntax or libraries, you can run Python's interactive interpreter. To make sure you have all the same eggs and packages available as Zope will when it starts up, run:

  $ ./bin/zopepy
  >>>

Write Tests for Everything

Finally, the biggest favor you can do yourself as a developer is to learn to write unit tests. To learn more about tests, see http://plone.org/documentation/tutorial/testing. Without unit tests, there is no telling whether your code is working, and whether you have inadvertently broken it. Since tests are normally small and isolated, they are also a great starting point for diagnosing problems or examining the behavior of other components. Try writing a small test and placing a pdb break point in it, stepping into other code as necessary.

Summary

In this article, we have seen:

• The various elements that make up a Zope installation, including the meaning of the $SOFTWARE_HOME and $INSTANCE_HOME environment variables
  
• Some background on Python's setuptools and eggs, and how they differ from Zope products
• How to use paster and zc.buildout to automate the creation and building of Zope instances in a repeatable manner
  
• Some important debugging tools, including pdb, DocFinderTab, and Clouseau
• A few tips on how you can more effectively help yourself by learning to look at source code, using pdb effectively, and prototyping things with zopepy

Plone development fundamentals Customizing Plone Developing new functionality Real-world deployments For more information, please visit: http://www.packtpub.com/Professional-Plone-web-applications-CMS/book