Home Web Development Professional Plone 4 Development

Professional Plone 4 Development

By Martin Aspeli , The Plone Foundation Alex Limi Toby Roberts (Project)
books-svg-icon Book
eBook $39.99
Print $65.99
Subscription $15.99
$10 p/m for first 3 months. $15.99 p/m after that. Cancel Anytime!
What do you get with a Packt Subscription?
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with video?
Stream this video
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
What do you get with Exam Trainer?
Flashcards, Mock exams, Exam Tips, Practice Questions
Access these resources with our interactive certification platform
Mobile compatible-Practice whenever, wherever, however you want
BUY NOW $10 p/m for first 3 months. $15.99 p/m after that. Cancel Anytime!
eBook $39.99
Print $65.99
Subscription $15.99
What do you get with a Packt Subscription?
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with video?
Stream this video
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
What do you get with Exam Trainer?
Flashcards, Mock exams, Exam Tips, Practice Questions
Access these resources with our interactive certification platform
Mobile compatible-Practice whenever, wherever, however you want
About this book
A years of experience in Plone development and project management are combined with an approachable writing style to create an engaging and highly-informative guide to working with Plone. Professional Plone 4 Development stands as an excellent resource for developers of all levels. - Eric Steele, Plone Release Manager Plone is a web content management system that features among the top 2% of open source projects and is used by more than 300 solution providers in 57 countries. Its powerful workflow system, outstanding security track record, friendly user interface, elegant development model and vibrant community makes Plone a popular choice for building content-centric applications. By customising and extending the base platform, integrators can build unique solutions tailored to specific projects quickly and easily.If you want to create your own web applications and advanced websites using Plone 4, Professional Plone 4 Development is the book you need.The https://www.packtpub.com/Professional-Plone-web-applications-CMS/book first edition of this book remains one of the most widely read and recommended Plone books. This second edition is completely revised and up-to-date for Plone 4.1, covering new topics such as Dexterity, Diazo, jQuery and z3c.form, as well as improved ways of working with existing technologies such as Buildout, SQLAlchemy and the Pluggable Authentication Service. It retains the writing style and comprehensive approach that made the first edition so popular. Built around a realistic case study, Professional Plone 4 Development will take you from an understanding of Plone’s central concepts, through basic customization, theming, and custom development, to deployment and optimization. The book is divided into four sections: First, you will be introduced to Plone and the case study, and learn how to set up a development environment. The second section covers basic customization, including theming a Plone site using Diazo. The third section focuses on custom development – building new content types and user interfaces, customizing security and integrating with external databases. The final chapters cover deployment and performance optimization.
Publication date:
August 2011
Publisher
Packt
Pages
516
ISBN
9781849514422

   

Chapter 1. Plone in Context

Since its humble beginnings, Plone has grown from "just another open source Content Management System" into a platform from which many people make their living, with hundreds of service providers worldwide. Big and small businesses, NGOs and charities, governments and individuals are building websites, intranets, and specialized applications on top of Plone with great success.

Spurred by advances in Python and Zope, along with increased media attention and recognition, Plone has steadily improved over the past several years. Plone 2.1 (released in 2005) and Plone 2.5 (released in 2006) were largely focused on incremental improvements and laying the foundations for future leaps in functionality. Plone 3.0 (2007) brought not only important new features to put Plone on par with most "enterprise" web content management systems, but instigated a more mature development process that led to incremental improvements through versions 3.1 (2008), 3.2 (2009), and 3.3 (2009). Plone 4 sets a new benchmark for stability and scalability, improves end user experience with a new theme and new visual editor, and brings Plone up-to-date with the latest Python and Zope releases.

In this chapter, we will:

  • Consider how Plone fits in with the ongoing evolution of Zope, and how the Zope Toolkit and newer Python technologies are changing web development practices
  • Consider when Plone may be a good choice for solving your web development problems
  • Discuss why and how you may want to become a part of the Plone community

A brief history of Plone

Plone is an application that runs on the Zope 2 application server. By now over a decade old, Zope was probably the first open source application server. It started life as an environment in which power users could build web applications through-the-web, and was at one point the "killer application" for the Python programming language. It has inspired various other applications and frameworks, as well as improvements in Python itself.

Zope 2 turned out to be a good platform for building content management systems, and the Content Management Framework (CMF) was created to make this easier. The CMF changed the emphasis of Zope programming towards filesystem-based development and applications that could be more easily packaged up and redeployed. It also brought us "tools", "skins", and many other concepts fundamental to Plone.

Plone was born as a user-friendly CMS using the CMF. It was initially a night-time collaboration between Norwegian Alexander Limi and Texan Alan Runyan, but other Zope and Python developers soon began to contribute as well.

As the Plone project built ever more functionality on top of Zope 2, the Zope developers were working on a successor, to be named Zope 3. This project aimed to start with a clean slate, drawing on the experiences (and mistakes) of Zope 2 and CMF to create a next-generation web development framework.

The main premise of Zope 3 was that small, reusable, easily testable, and well-documented components should be orchestrated into complete systems. The individual building blocks should also be usable from other Python applications. Eventually, the community realized that Zope 2 could become just such an application. By way of a piece of integration code called Five (hint: what is Zope 2 + Zope 3?), it became possible to use Zope 3 components and concepts directly in Zope 2 applications, including Plone.

Zope 3 and Five revitalized Zope 2 development. Zope 2.8 shipped with the components from Zope 3.0 as an add-on library. Subsequent versions continued to update parts of the Zope 2 internals with more modern components, in tandem with the evolution of Zope 3 itself.

Unfortunately, this co-mingling of Zope 2 and Zope 3 quickly became confusing, and Zope 3 never really took off as a standalone framework in the same way that Zope 2 had, not at least because by this time there were so many other Python frameworks to choose from.

In 2010, the Zope community decided to rebrand the "core" part of Zope 3 as the Zope Toolkit, or ZTK for short. The ZTK is used by a variety of applications, including Zope 2, and is advanced jointly by the developers of those projects. The application server and framework aspects of Zope 3 were moved out to a separate project called Blue Bream, which thus became another consumer of the ZTK. The name Zope 3 was retired, although you will still see it used in older documentation and code.

What is new in Plone 4?

The first edition of this book covered Plone 3. This edition has been updated to take into account new and changed components that come with Plone 4, as well as evolution in Plone 'best practice' over the past few years.

Some of the more exciting new features in Plone 4 include:

  • Plone 4 uses Zope 2.12 (up from 2.10) and Python 2.6 (up from 2.4). Plone 4.1 updates Zope to 2.13.
  • Binary files and images are now stored in so-called BLOBs by default, making Plone much better at serving large files.
  • There is a new default visual theme, known as Sunburst.
  • The default visual content editor has been changed from Kupu to the widely used TinyMCE editor. This is both better maintained and easier to extend, and should provide an improved user experience for content authors.
  • Plone's core JavaScript functionality is now based on jQuery and jQuery Tools. The most obvious change is that many features are now presented in modal "overlay" dialog boxes.
  • Portlet management has been improved with a better user interface for "blocking" portlets and per-group dashboard portlets, in addition to per-user ones.

It is now possible to let users log in with their e-mail address instead of a Plone-specific username.

By the time you read this book, Plone 4.1 should be out. This includes many incremental improvements, such as:

  • A new Site Administrator role, for users who should be able to administer the site without having access to the lower-level developer- and system administrator-oriented tools in the Zope Management Interface.
  • Better support for commenting on content, including moderation and Catpcha support, based on the new plone.app.discussion package.
  • Inclusion of a new HTTP caching tool, plone.app.caching.
  • A new standard approach to assign and look up universally unique identifiers (UUIDs) for content and other objects, based on the plone.uuid package.
  • Integration into the core distribution of the form building tool z3c.form, which we will cover in Chapter 11, Standalone Views and Forms, the testing framework plone.testing, which we will use throughout the book, and plone.app.registry, which is used to manage site-wide settings and preferences. In addition, the lxml XML parsing library will be bundled with Plone's installers to simplify installation.
  • Factoring out of some dependencies, so that it is possible to deploy a minimalist Plone installation without such things as the KSS Javascript framework, the Kupu visual editor, and OpenID support. The installers and standard distribution will continue to include these, of course, but by depending on the Products.CMFPlone distribution instead of the fully fledged Plone distribution, we can choose to only include those add-ons we need.

For more details, please see http://plone.org/products/plone/releases/4.0 and http://plone.org/products/plone/releases/4.1.

Note

We will use Plone 4.1 as the basis for the examples in the book, but we will highlight where there are differences between Plone 4.0 and 4.1 and how you can stay compatible with both releases.

Plone-the-application and Plone-the-framework

New users sometimes appear on the Plone mailing lists asking for a comparison of Plone and PHP, the popular web programming language. On the face of it, this is a strange question, if you consider that PHP is a language and Plone is first and foremost a content management application. You can download Plone, put it on your server, configure a few options, perhaps install a third party add-on product or two, and use it to solve your everyday content management needs.

Thus, "Plone-the-application" is used to power intranets, public websites, document repositories, and a host of other web-based systems. Plone successfully competes in the "upper mid-tier" CMS market, and is often chosen over commercial systems such as OpenText RedDot and Microsoft SharePoint in head-to-head evaluations.

Plone is developed almost exclusively by volunteers. It is open source, which means that you can obtain and use it for free, and that you are free to modify the underlying source code. There is no official, commercial "high end" version. There is no single company behind Plone selling support or certifications (although professional support is available from a multitude of smaller vendors). There is no specific hardware tie-in. So why have thousands of man-hours gone into making Plone an ever more sophisticated CMS?

The answer is two-fold. We will consider the community drivers later in this chapter, but there are strong commercial reasons as well. The majority of Plone's core contributors make their living from what is often referred to as "Plone-the-framework". They are professional web developers who sell consulting services and support, and have found that by working off an advanced, common base platform, they can offer better value to their customers. A few Plone contributors work for companies with large Plone installations that have found paying someone to spend part of their time contributing to Plone and getting changes into the core to be cheaper than buying ad-hoc support and development resources.

This model is of course nothing new in open source, but it happens to fit content management systems quite well. Customers rarely need a CMS as it comes out of the box. Most will want to customize its look-and-feel, workflow, security, and site structure. Frequently, customers also need some integration with existing systems, or may wish to build a portal that includes not only web page content, but various interactive tools and mini-applications.

If a customer is going to pay for consultants or developers to create the system they need, buying an expensive software license as well seems unnecessary. Developing a complete system from scratch is normally prohibitively expensive and risky. Better then, to take a system which comes close to meeting their needs, turn off the parts that are not relevant, and add the pieces that are missing. That is where Plone comes in. Customers can also take comfort from the fact that there is a large community of companies and individuals who can work with Plone. They need not be locked into a single vendor.

Because the people who build Plone spend the rest of their time building these more specialized systems on top of it, Plone's architecture has evolved in such a way that it is easy to extend. Indeed, this kind of extensible application is how Zope 2 (the application server on which Plone runs) was originally marketed. Almost any part of Plone can be amended, changed, or modified in such a way that Plone itself can be upgraded later without needing to reapply changes to the actual code base. That is, you should never have to fork Plone for your own needs.

Additional considerations when deciding on Plone

Whether Plone is a good base for your application or not will depend on how much Plone offers you out of the box, and how difficult it will be to provide the rest. Usually, this means that your requirements can be modeled in a "content-centric" way, making use of Plone's infrastructure for managing hierarchical, semi-structured content. Being able to reuse Plone's workflow-based security model, tried-and-tested user interface and its infrastructure for things like user management, page layout, and administration tasks also tends to be a strong selling points. Furthermore, Plone's strong support for internationalization and localization is an important consideration for many users.

At the same time, it is important to realize that to get the most out of Plone, you will need to make an investment of time, money, or both. Zope and Plone are RAM-hungry and run best on a modern server. Proper infrastructure is never free, and requires some planning. Similarly, if this is your first Plone project and you are intending to customize Plone extensively, you should bear in mind that there will be a learning curve. Besides online documentation and books such as this one, various companies also offer training courses and specialized consultancy, should you need it.

Licensing

Most parts of Plone are licensed under the GNU General Public License (GPL) Version 2, with various components alternatively licensed under the Lesser General Public License (LGPL), MIT, and BSD licenses. You should seek qualified legal advice if you are concerned about the license.

In practical terms, the license means that you are free to modify and reuse parts of Plone for your own needs. However, if you build a custom application on top of Plone and you intend to distribute (for example, sell a license for, or a boxed version of) that application, you will need to distribute its source code as well. You do not need to make the source code available if you are simply deploying a solution on an internal server.

Plone's source code is legally owned by the Plone Foundation, and is protected by a contributor agreement drawn up with the aid of the Software Freedom Law Center. This "software conservancy" model is very similar to the framework used to protect the integrity of other major open-source projects, such as Apache and Mozilla. The Plone Foundation is able to negotiate alternative license arrangements in special circumstances. Please see http://plone.org/foundation.

Licensing

Most parts of Plone are licensed under the GNU General Public License (GPL) Version 2, with various components alternatively licensed under the Lesser General Public License (LGPL), MIT, and BSD licenses. You should seek qualified legal advice if you are concerned about the license.

In practical terms, the license means that you are free to modify and reuse parts of Plone for your own needs. However, if you build a custom application on top of Plone and you intend to distribute (for example, sell a license for, or a boxed version of) that application, you will need to distribute its source code as well. You do not need to make the source code available if you are simply deploying a solution on an internal server.

Plone's source code is legally owned by the Plone Foundation, and is protected by a contributor agreement drawn up with the aid of the Software Freedom Law Center. This "software conservancy" model is very similar to the framework used to protect the integrity of other major open-source projects, such as Apache and Mozilla. The Plone Foundation is able to negotiate alternative license arrangements in special circumstances. Please see http://plone.org/foundation.

The Plone Community, and why you belong there

The word "community" permeates any discussion of what Plone is and where it came from. In practical terms, Plone may be a piece of software, but in the truest sense, Plone is first and foremost a community. In the words of former Zope Corporation CEO Paul Everitt:

"Plone, the software is an artifact of Plone, the community."

Almost all of Plone's core contributors are (or become) friends in real life. They arrange "sprints"—short, intense development sessions—sometimes in exotic locations like an Austrian castle, a former military base on a Norwegian island, and a cabin high up in the Alps. There is an annual conference and usually a symposium or two throughout the year. And every day, the developers meet in online chat rooms and on mailing lists.

This friendship and the mutual respect that developers have for each other, are important factors contributing to Plone's success. Many of us care quite passionately about making Plone the best it can be, and happily expend both personal and professional time on Plone-related activities without direct financial reward.

The Plone community itself is larger than just the two or three dozen core developers, though. First, Plone's sister communities—those of Zope, CMF, and Python—overlap with the Plone community and with each other in socially complex ways. Second, a large number of developers contribute third-party add-on products, answer questions from end users and other developers, and participate in discussions around the future of Plone. A larger number still are end users and Plone administrators, reporting bugs, offering praise and criticism, and joining in the discourse. This is where we hope you will connect with the community initially, if you have not done so already!

Most open source communities have a core of dedicated developers with some governance structure around it. In Plone's case, governance is provided by:

  • The Plone Foundation, which is responsible for legal affairs and has a mission "to protect and promote Plone."
  • The current Release Manager, who has the final say over what goes into a particular release. A release manager typically serves for one or two releases, before handing over the reins (and serving as a mentor) to the next release manager.
  • The current Framework Team, who reviews contributions and make recommendations to the release manager during the early stages of the release cycle.

In practical terms, however, Plone's governance is extremely democratic, and there is very little conflict and very few emotional disputes.

Because of this, people generally find the Plone community open and approachable. Most developers are very happy to give help to those who ask for it, and questions on the mailing lists and in the chat room (see http://plone.org/support) are normally answered quickly. Many developers will also actively seek to involve more peripheral members of the community in improving Plone, for example through mentoring or invitations to sprints and other events.

One of the best qualities of the Plone community is its openness to new contributors and the deliberate way in which it develops new leadership from within. The users and developers which encircle the core will sometimes move closer to it through their own learning and participation. As they gain the trust and respect of other developers, they are given more decision-making powers and less supervision, and will be able to influence the future direction of Plone more directly.

Such influence is one strong benefit of actively engaging with the community, and it is not as difficult to attain as one might think. The main factor is attitude, not knowledge. For example, there are many people close to the core of the project who are less technical, but who want to help where they can. Ultimately, Plone would not survive without an influx of fresh blood and new perspectives from time to time.

Even if you are not enticed by rewards of responsibility and influence, being a member of the Plone community, however peripheral, will almost always be beneficial. By reading the mailing lists, for example, you will pick up much up-to-the-minute knowledge that may not be readily available elsewhere. When you are stuck, asking a question in the chat room or on a mailing list can often get you an answer in minutes or hours. By meeting Plone users and developers in real life, at user group meetings, sprints, conferences, and symposia, you will find yourself with a growing network of experts to draw upon when you need it the most. Save for Alan and Alexander, who started it all, every one of us was once a Plone newbie—many of us more recently than you might think!

Contribution to the community should be fun, fit your skills and interest, and give you something back. The easiest way to make a contribution is simply to start answering questions on the mailing lists. If you have some code you want to write, ask about how it may fit with existing projects, and how you may best contribute it to the community. If you feel there is a gap in the documentation and you would like to write a how-to or tutorial, you can do so at http://plone.org/documentation and submit it for review. If you would like to host a user group meeting or a sprint, get in touch! You will find that if you show a willingness to give a little, you will get a lot.

Summary

In this chapter, we have discussed:

  • A brief history of Zope and Plone
  • New features in Plone 4
  • How Plone-the-application and Plone-the-framework are related
  • Some considerations you should bear in mind when deciding to use Plone
  • What the Plone community is and why you may consider knocking on its door

In the next chapter, we will introduce the example application, which will be used to illustrate key concepts over the course of the book.

 

Chapter 2. Introduction to the Case Study

Throughout this book, we will build a semi-realistic application that will demonstrate various Plone technologies and concepts. The source code for this application can be found on the book's accompanying website. We will explain the various packages and modules over the next several chapters, but if you are the type of reader, who likes to start at the end, feel free to browse through the code now.

In this chapter, we will:

  • Put the example application in the context of a case study
  • Show our fictitious client's requirements
  • Perform some high level modeling of what the application may look like in Plone.

Background and high-level requirements

Optilux Cinema Group is a mid-sized chain of cinemas. They currently have a limited web presence, but wish to expand it to offer moviegoers a better way to find out about the latest films and reserve tickets for screenings.

The following high-level requirements have been presented to potential vendors of the cinema's web content management system:

 

Requirement

Importance

Chapter

1

The site should have a look and feel consistent with Optilux's corporate branding.

High

5 and 8

2

The site should show information about all of Optilux's cinemas.

High

10

3

Non-technical cinema staff should be able to update information about each cinema.

High

6 and 10

4

The site should allow staff to highlight promotions and special events. These may apply to one or more cinemas.

High

10

5

Cinema staff should be able to publish information about new films. It should be possible to update this information after publication.

High

6 and 10

6

Customers should be able to find out in which cinemas a particular film is showing, and which films are showing at a particular cinema. Note that the scheduling of films at cinemas is managed in an existing relational database.

High

12

7

Only films that are currently showing or will be shown in the near future should be viewable.

High

10

8

Customers should be able to search and browse films by cinema, location, and date/time or film name.

High

10 and 12

9

Customers should be able to reserve tickets online. Tickets will be picked up and payment taken at the cinema. Reservations must use Optilux's existing relational database-based ticketing system.

Medium

12

10

Any digital assets, such as images or other files, used on the website must be traceable using a special Digital Asset Management Code, which will be used to link the assets to a third party system.

Medium

10 and 11

11

Customers should not have to log in to use the site, but a username and password should be required when they wish to reserve tickets.

Medium

12 and 13

12

Logged-in customers should have easy access to their preferred cinema or cinemas, such as those in their area.

Medium

13 and 14

13

Customers should be able to log in using their Facebook account.

Medium

13

14

Customers should be able to email enquiries to the site manager if they have questions or feedback.

Low

11

15

Customers should be able to discuss and rate movies.

Low

14

As we become more experienced with Plone development, these requirements will start to ring a few bells. For example, we may identify content types by finding the nouns in the requirement descriptions (for example: #2, #3, #4, and #5), such as Cinema and Film. We may be able to satisfy a few requirements by using Plone's standard content types or simple extensions thereof – a Promotion (#4) could be an extension of an Event or News Item.

It is also likely that the various content types will require custom workflows and security (#5, and #7). We can identify user roles like Customer, Staff, and Management from the subjects in the various requirement descriptions and start to understand what permissions these roles may have.

For reservations and reporting, we will need some relational database connectivity (#9). This in turn will probably mean developing custom forms and templates which access the information in the database.

As the system requires management of member (user), data, and preferences (#11, and #12) we may need to add additional user properties. To support an external authentication mechanism such as the Facebook Graph API (#13) we need to write a custom plugin for the Zope Pluggable Authentication Service (PAS).

Lastly, we must provide client-specific branding (#1). Plone provides user-friendly administrative pages and content management operations. We may also want to use the jQuery JavaScript framework to add dynamic user interface elements.

Modeling and mockups

We may, perhaps in conjunction with the client, decide to do some initial modeling of how the system should look. Some developers advocate very detailed modeling and strict adherence to relevant standards such as UML, the Unified Modeling Language. This depends on personal preference. In the author's opinion, the models themselves are not as important as the act of modeling—thinking through the client's requirements in terms of high-level components and interfaces.

Models can help structure a conversation with the client. Walking through a model and showing screen mockups or wireframes brings an abstract idea of an application to life.

For the purpose of this application, we will start by drawing a UML Use Case Diagram, which highlights the main types of users of the system and the types of things they may want to do with it:

Modeling and mockups

This diagram is by no means exhaustive, but it does give a quick visual representation of the kind of interaction the application will need to support.

From this, we can identify major system components, represented in a high-level class diagram. This also shows where external interfaces will be needed, in this case to the relational database holding film reservations.

Modeling and mockups

Again, this diagram is far from exhaustive, but it shows that we need to manage Cinema and Film, which is related by Screening. Customer and Staff are different types of site users, and Customer can make Reservation for particular Screening. Customer can also create Rating for a particular Film. In the content collaboration system, Staff can be members of various Projects, which contain documents that staff are working on.

We will also typically work with visual design mockups, drawn in a graphics program and/or provided as 'flat' HTML pages, to indicate how the site will look and feel:

Modeling and mockups

In Chapter 8, Creating a Custom Theme, we will see how this mockup forms the basis of a custom site theme.

Information architecture

The design and requirements have implications for the site's information architecture. In Plone, content authors add folders, pages, and other content items to build site structure. This structure is automatically reflected in Plone's navigational aids—the top level tabs, breadcrumbs, navigation tree, and site map, that is, in the website's URLs.

Tip

A logical site structure is an important step towards making a website user-friendly and easy to navigate, and can help improve search engine rankings.

Looking at the design mockup, the links across the top and in the left hand side navigation will most likely expose the main functionality of the site. Different options may appear for logged-in customers, cinema staff or managers, as appropriate. Note that the labels in the mockup are suggestions from the graphic designer. The actual information architecture may be different and may change over time, depending on how the site administrator wishes to set it all up.

The following table outlines initial information architecture for the site:

Section

Purpose

Content types

Visibility

Home

Front page, contains general information

Folder, Page

All

About Optilux

General information about Optilux Cinemas

Folder, Page

All

About membership

Information about membership benefits

Folder, Page

All

Films

Container for films, including film listing

Film Folder

All

First film

Information about a film currently showing

Film

All

Unpublished film

A film not yet visible to the public

Film

Staff

Cinemas

Container for cinemas, including listings

Cinema Folder

All

Region one

Container for cinemas in one region

Cinema Folder

All

Promotion

A promotion specific to Region one

Promotion

All

Cinema one

A cinema in Region one

Cinema

All

Screening times

Listing of screening times for films

Screening

All

Reserve

Form to reserve tickets for a screening

Reservation

Member

Cinema two

Another cinema in Region one

Cinema

All

Promotion

A promotion specific to Cinema two

Promotion

All

Future promotion

A promotion not yet visible

Promotion

Staff

Region two

Container for cinemas in another region

Cinema Folder

All

Corporate information

General corporate information

Folder, Page

All

Project one

A project, visible to all staff

Folder

Staff

Project two

Another project, private to one group

Folder

Some staff

This structure should translate reasonably to a site map of the content hierarchy. Typically, a folder may represent a section or subsection, and contain pages or other content items as necessary. Most folders will have a 'front page' set as its default view, using Plone's Display menu, rather than use the standard folder listing view.

This pattern is also used for custom content types. For example, we will create a Cinema Folder type to contain Cinemas. The default view of the cinema folder will allow users to browse for cinemas. Custom types will be built in Chapter 10, Custom Content Types.

Note

It is often helpful to prototype the site's information architecture using standard content types (mainly Folders and Pages) with placeholder text, even if some items will eventually be built using custom types. This makes it easier to test the navigation and any customizations.

Running the project

Project management is outside the scope of this book, and every developer and organization will have different ways of managing their development projects. It is probably fair to say, however, that most Plone consultants prefer to work according to "agile" principles (See http://agilemanifesto.org), which typically include rapid prototyping, short development iterations and a high degree of customer interaction throughout the design and development process.

Indeed, Python is often branded an 'agile' programming language, because it is quick to write, easy to read and lends itself well to code refactoring. Furthermore, since Plone gives you a fully featured system out-of-the-box, you will be able to have something tangible and usable up and running almost instantly, which you can incrementally refine with input from the client.

Many developers keep a live test server for the client to test, periodically deploying code updates to showcase new functionality. If you go down this route, you will probably want to set up an issue tracker for the client to report any problems they find. There are many options here. Trac (http://trac.edgewall.org) is a popular Python-based test server.

Getting a draft of the customer's desired branding up for testing quickly can be very valuable, as users often have trouble making the conceptual leap from vanilla Plone to their own requirements. Even just putting a logo and color scheme into a site otherwise using the standard Plone theme can give the client a greater sense of ownership and generate some excitement. It is normally advisable to turn off unneeded core Plone functionality sooner rather than later, to avoid initial confusion. Also remember that in general, you cannot assume that delivering something Plone does out of the box will involve zero effort. At the very least, you will have to test the functionality within the overall context of your application.

As you work with your client, you will get a better idea about what is important to them. Giving them what they want most first, getting them involved in testing early, and being receptive to changes in the specification is usually a very good idea.

Because of this, many Plone consultants prefer to work on a time-and-expenses basis and promise frequent code releases, allowing the project to grow with the client's understanding of their needs. This is not so different from how Plone itself grows as people discover new things they would like it to do. By following good development and project management practices and effectively leveraging Plone's base functionality, you can ensure that at any given point the client has a fully functional (but partially complete) application at their disposal for testing and experimentation.

Summary

In this chapter, we have been introduced to:

  • A semi-realistic case study that will be used throughout this book
  • High level requirements for the example application following this case study
  • Some initial models that point the way to how the application will look and work
  • Initial thoughts on the site's information architecture
  • A few tips for running Plone development projects

In the next chapter, we will learn how to set up our development environment, before we start building the application for real.

 

Chapter 3. The Development Environment

Before we can start building our application, we must set up a suitable development environment. This should mirror the final 'live' server configuration as closely as possible, so that we can test our software locally before deploying it. The environment should also provide appropriate tools to support us during development.

In this chapter we will learn about:

  • The elements of Zope's software stack
  • How these can be configured for development on various platforms
  • Supporting technologies, such as distributions, Setup tools/Distribute, and Buildout
  • Some useful development tools

Installers

Understanding your development environment is an important prerequisite to becoming a productive developer. If you need to get up and running quickly, however, you can start with one of the prepackaged Plone installers.

Note

The Plone community generally discourages the use of third-party operating system packages such as RPMs or DEBs for installing Zope and Plone. These are often out of date, and sometimes make non-standard assumptions that make them difficult to support. Furthermore, as a developer you will need write access to the build configuration and source code of your own packages. As such, you should run Zope from within your home directory. We will cover production server deployment in the final part of this book.

Since Plone 3.2, the various platform-specific installers have been based on Buildout, Plone's preferred build tool. You can download the latest Plone 4 installer for your platform from http://plone.org/downloads.

The rest of this chapter will describe the building blocks of a development environment and illustrate how to set up such an environment from scratch. If you use an installer, the result may be slightly different, but once you understand how Buildout works, you will be able to customize the installer-configured environment as necessary.

Note

Note that on Mac OS X, the Unified Installer is the best choice for developers, as it is more Buildout-friendly.

Glossary of terms

The table below summarizes the various terms and technologies which we will encounter in this chapter. It pays to be familiar with these, as you will find them again not only throughout this book, but also in other Plone documentation.

Term

Definition

Module

A single .py file of Python code.

Package

A bundle of Python modules and supporting files. At the most basic level, a package is a directory with an __init__.py file and zero or more other Python modules. Packages may be nested. See the Python documentation for details about how to create and import packages.

Setuptools / Distribute

Setuptools is a Python library that extends Python's built-in distutils library to support extended distribution metadata, including dependency management (see below).

Distribute is a fork of Setuptools that provides some additional features and bug fixes. At the time of writing, it is the more actively maintained of the two, and the default choice for Plone 4.

Namespace package

Nested packages can be used to build namespaces. For example, most Plone software lives in a top-level package called plone, for example plone.browserlayer and plone.memoize. This is useful to avoid clashes with other projects and to signal ownership over a particular package.

Namespace packages, a feature of Setuptools/Distribute, allow such namespaces to be declared explicitly. This in turn makes it possible to release packages independently: plone.browserlayer and plone.memoize are independent distributions on PyPI (see below), which can be installed independently and/or concurrently.

Distribution

A means of packaging and distributing Python code. A distribution consists of a directory with a setup.py file, Python code, and potentially other resources. The metadata in setup.py may include a version, installation dependencies and license information.

Tools based on Setuptools/Distribute, including Buildout, can use the metadata to manage concurrent installations of different versions of a given distribution, automatically fetch dependencies, and more.

It is common to create a distribution containing, and named after, a single package. For example, the distribution plone.memoize contains the package plone.memoize (which may of course contain sub-packages). However, distribution names do not have to follow this pattern.

Egg

A binary distribution format used by Setuptools/Distribute. Each distribution archive is specific to a particular platform and Python version. Therefore, it is normally safer to use a source format when distributing packages, which is effectively just a compressed archive of the code and metadata. (Setuptools/Distribute will turn a source distribution into an egg specific to the local development environment upon installation.)

The downside of using source distributions is that if a package has binary dependencies (for example Python extensions written in C), the necessary compiler and libraries need to be available wherever the distribution is installed. It is common to assume Linux and Mac OS X users have these tools available, but make binary eggs available for Windows.

Develop egg

Normally, we let Setuptools/Distribute (via Buildout) manage the downloading and installation of distributions. For packages we are developing, however, we normally want to control the source code ourselves, and have the abilityto have changes reflected immediately without having to rebuild and reinstall a distribution. This can be achieved with develop eggs, which use a placeholder file to inform Setuptools/Distribute about the location of the in-development source code.

Product

Filesystem-based Python extensions for Zope 2 were originally called Products. Placed in a special directory (Products/), they would automatically be discovered by Zope at startup, and implicitly appear under the Products.* namespace. Although this method of deployment is still supported for legacy software, virtually all products have now been converted to regular packages.

Products may contain an initialize() function, called when Zope starts up. For legacy reasons, this function is called automatically for packages in the Products.* namespace. Packages in other namespaces can opt into this behavior using the <five:registerPackage /> directive in their configure.zcml file. This is mainly useful for registering new content types. We will see an example of this in Chapter 10, Custom Content Types.

The term "product" is also sometimes used more generally to refer to add-ons that extend Plone, even if they are not actually packaged as old-style Zope 2 Products.

Python path

The Python path lists directories and distributions containing Python packages that are available for import by a running Python program. The current path is stored in the variable sys.path. We will use Buildout to ensure that this path contains the packages we need to run Plone and our custom software, and – ideally – no more.

The Python Package Index (PyPI)

Previously known as the "Cheese Shop", this is an online repository of Python distributions. Anyone can upload a distribution here. Setuptools/Distribute-based tools such as Buildout and easy_install can use PyPI to locate and install distributions and their dependencies.

easy_install

A command-line tool installed with Setuptools/Distribute, which searches PyPI for a given distribution and downloads and installs it.

By default, easy_install puts packages in the global site-packages folder for the Python interpreter which was used to install easy_install itself. This, coupled with limited dependency version management features, makes easy_install unsuitable for installing Plone and Zope.

Pip

A more advanced alternative to easy_install, which can better manage dependency versions and supports distribution uninstallation. This is rarely used in Plone development (and unsuitable for installing Plone itself), but useful for installing tools written in Python.

Buildout

A tool for managing a self-contained environment through a single configuration file (buildout.cfg). Buildout itself is installed by the zc.buildout distribution. We also use the noun 'buildout' (lowercase 'b') to describe a particular environment managed by Buildout (capital 'B').

Most Plone projects use Buildout to download, install, and configure Zope, Plone, and other dependencies. Buildouts are 'repeatable', meaning that they can be used to replicate a particular setup across multiple servers or developers' machines.

Known Good Set (KGS)

Zope and Plone consist of hundreds of distributions, all available on PyPI. These evolve semi-independently—new versions may be released at any time.

By default, Buildout, easy_install, and Pip will download the most recent distributions of a package and its dependencies. This may be fine for simple projects, but for complex systems such as Zope or Plone, there is no guarantee that the latest versions of all dependencies will work together. Furthermore, a new release of a dependency could make a 'stable' build difficult to replicate in the future.

To avoid such problems, the Zope and Plone developers publish 'known good sets' that 'pin' every distribution to a tested version. Custom projects usually start from a standard KGS, and then override or amend the version pins as necessary to accommodate additional software.

Virtualenv

Just as stray versions can destabilize an installation, so can conflicting packages installed globally into the Python interpreter's site-packages directory, e.g. using a tool like easy_install or Pip. Many operating systems also ship with Python installations that contain operating-system specific packages, some of which have been known to conflict with Zope and Plone.

Virtualenv is a script that can be used to create a clean, isolated Python installation. It also allows distributions to be installed in a way that is local to this environment, so that they do not affect other installations.

ZopeSkel

A collection of Paste Script templates for Zope and Plone development, accessible through the zopeskel command. We will use this to create new package distributions.

Prerequisites

Before we can get into the business of setting up a Plone development environment, we must install a few prerequisites:

  • Python interpreter: This is suitable for running Plone. As of Plone 4.0, that means Python 2.6. If your operating system does not have Python 2.6 installed, you can get the latest version in the 2.6 series from http://python.org/. On Mac OS X, you can also use MacPorts (http://www.macports.org/) to install it. Some members of the Plone community have also created a cross-platform Buildout that can be used to build Python itself. This is particularly popular for installing Python 2.4 (used in Plone 3) on Mac OS X Snow Leopard. See http://svn.plone.org/svn/collective/buildout/python.
  • Python Windows Extensions (On Windows only): It is an installer suitable for your Python version and architecture can be downloaded from http://sourceforge.net/projects/pywin32/files/. Presuming you installed Python to C:\Python26, you should add C:\Python2.6;C:\Python2.6\Scripts to your system PATH.
  • Python Imaging Library (PIL): This should be installed for this Python interpreter. Download it from http://www.pythonware.com/products/pil. This may have other dependencies, such as the JPEG, PNG, and GIF libraries. On Mac OS X, you can use MacPorts to install PIL, if you also used MacPorts to install Python 2.6. On Windows, there is a PIL binary installer. On other operating systems, you may want to install the development versions of libjpeg, libpng, and libgif using operating system packages, and then install PIL manually from source. Simply unpack it and run python setup.py install from inside the newly unpacked directory, where python is the binary you intend to use to run Plone.
  • C compiler (On Unix-like operating systems): You will need this to compile Zope and its dependencies. The default choice is GCC. On Mac OS X, this means installing XCode with command line tools from the Mac OS X DVD. On Linux, use your operating system's package manager to install the base GCC package. On Windows, there should be binary distributions available for all dependencies, making a compiler unnecessary.
  • An internet connection: Buildout needs to download and install Plone. If your connection requires a proxy server, Python should respect the operating system proxy settings. If it does not, you may need to set the HTTP_PROXY environment variable to the proxy server's URL.
  • A programmer's text editor: The editor should be preferably one with Python, XML, HTML, and CSS syntax highlighting. You should set up your editor, so that a tab or indent is output as four (4) spaces. This makes Python development a lot more predictable. Popular choices include TextMate on Mac OS X, vi or Emacs on Linux, and SciTE on Windows.
  • A shell: Most examples in this book will show a Bash interpreter shell, though we will cover Windows syntax when it differs significantly. Remember, that path separators on Windows are backslashes (\), while other environments use forward slashes (/). Also, environment variables on Windows are referred to as %NAME%, whilst in most Unix shells, including Bash, variables are dereferenced with $NAME.

Note

This book assumes that you are comfortable using the shell to move around the filesystem, run commands, and manipulate files; and that you know how the PATH environment variable works and how to change it.

Creating an isolated Python environment

If you installed or compiled a version of Python specifically for the purposes of Plone development, and you did not install anything else globally, you can skip this step.

Note

This is likely to be the case if you are on Windows (see the next section), which is just as well, as it is tricky to install PIL into a Virtualenv on Windows.

If, however, you want to use an operating system-provided Python installation, or you intend to install anything into the global Python environment, you should create an isolated Python environment for Plone development using Virtualenv by executing the following steps:

  1. First, make sure easy_install is available. You can try this on a command line with:
    $ easy_install -h
    
  2. If this prints a help message, all is good. Otherwise, download distribute_setup.py from http://python-distribute.org/ and run it with:
    $ python distribute_setup.py
    

    Note

    If you get a permission denied error, you may need to run this command as root or under sudo.

  3. You should then be able to install that latest version of Virtualenv using easy_install:
    $ easy_install -U virtualenv
    

    Note

    Again, you may need to do this as root or under sudo. If the command is not found, look at the output of the distribute_setup.py command to find out where easy_install was configured, and add this to your PATH environment variable.

  4. At this point, you should have a virtualenv command available. Use this to create a new environment, such as in your home directory. For example:
    $ virtualenv --no-site-packages plone-python
    

    Note

    This will use the same Python interpreter as the one used by easy_install. If you want to use a different Python binary (for example: if easy_install does not run under Python 2.6), you can specify the full path to the python binary with the -p command line option.

This will install a 'clean' Python binary in plone-python/bin/python. The --no-site-packages ensures no global site packages 'bleed' into the environment.

  1. This unfortunately also hides PIL, so you will need to install the Python components of this again. Download the latest version from http://pythonware.com/products/pil/. Unpack it to reveal the directory Imaging-1.1.7 (assuming 1.1.7 is the version you downloaded). Then do:
    $ cd /path/to/Imaging-1.1.7
    $ /path/to/plone-python/bin/python setup.py install
    

    Note

    Please note that at the time of writing this book, PIL is not compatible with setuptools and thus cannot be installed with easy_install or Buildout.

Creating an isolated Python environment

If you installed or compiled a version of Python specifically for the purposes of Plone development, and you did not install anything else globally, you can skip this step.

Note

This is likely to be the case if you are on Windows (see the next section), which is just as well, as it is tricky to install PIL into a Virtualenv on Windows.

If, however, you want to use an operating system-provided Python installation, or you intend to install anything into the global Python environment, you should create an isolated Python environment for Plone development using Virtualenv by executing the following steps:

  1. First, make sure easy_install is available. You can try this on a command line with:
    $ easy_install -h
    
  2. If this prints a help message, all is good. Otherwise, download distribute_setup.py from http://python-distribute.org/ and run it with:
    $ python distribute_setup.py
    

    Note

    If you get a permission denied error, you may need to run this command as root or under sudo.

  3. You should then be able to install that latest version of Virtualenv using easy_install:
    $ easy_install -U virtualenv
    

    Note

    Again, you may need to do this as root or under sudo. If the command is not found, look at the output of the distribute_setup.py command to find out where easy_install was configured, and add this to your PATH environment variable.

  4. At this point, you should have a virtualenv command available. Use this to create a new environment, such as in your home directory. For example:
    $ virtualenv --no-site-packages plone-python
    

    Note

    This will use the same Python interpreter as the one used by easy_install. If you want to use a different Python binary (for example: if easy_install does not run under Python 2.6), you can specify the full path to the python binary with the -p command line option.

This will install a 'clean' Python binary in plone-python/bin/python. The --no-site-packages ensures no global site packages 'bleed' into the environment.

  1. This unfortunately also hides PIL, so you will need to install the Python components of this again. Download the latest version from http://pythonware.com/products/pil/. Unpack it to reveal the directory Imaging-1.1.7 (assuming 1.1.7 is the version you downloaded). Then do:
    $ cd /path/to/Imaging-1.1.7
    $ /path/to/plone-python/bin/python setup.py install
    

    Note

    Please note that at the time of writing this book, PIL is not compatible with setuptools and thus cannot be installed with easy_install or Buildout.

A minimal buildout

With the prerequisites out of the way, we can now install Zope and Plone using Buildout. Later in this chapter, we will create a flexible development buildout, which we will use and extend throughout the book. First, however, we will illustrate a minimal Plone buildout and show how to stop and start Zope in debug mode.

  1. First, we create a new directory to contain the buildout somewhere we have write access as a non-administrative (root) user, such as our home directory. It is preferable to use a short directory path, without spaces, as some older tools do not deal well with spaces in file paths.
  2. Next, we download bootstrap.py, which is used to install Buildout itself. It is best to download the version used by the Plone developers: http://svn.plone.org/svn/plone/buildouts/plone-coredev/branches/4.1/bootstrap.py (adjust the version as necessary for Plone 4.2 and onward).
  3. We place this file in our newly created directory. In the same directory, we use a text editor to create a buildout.cfg file with the following contents:
    [buildout]
    extends =
        http://dist.plone.org/release/4.1/versions.cfg
    parts = instance
    
    [instance]
    recipe = plone.recipe.zope2instance
    user = admin:admin
    eggs = Plone

    Note

    The extends line is used to include a 'known good set' of distributions corresponding to a particular Plone release. You should adjust the version number (4.1 in this case, which is the latest release at the time of writing) as necessary.

  4. To initialize the buildout, open a shell and move to the directory containing the two files. Then we run the following command line which will download Buildout and create a few directories and files.
    $ python bootstrap.py --distribute
    

    Note

    If you want to use a different Python binary to run Plone, you can invoke one by absolute path. For example, using the Virtualenv example (shown earlier), you could run /path/to/plone-python/bin/python bootstrap.py --distribute.

  5. We can then run:
    $ bin/buildout

    This will download, compile, and install Zope and Plone. The first time we do this, it can take a long time.

    Note

    You may also see some warnings about 'syntax errors' fly past when Setuptools/Distribute tries to pre-compile certain Python scripts. You can safely ignore these. They are issued because, these scripts are not normal Python modules, but rather scripts intended to be executed in Zope's untrusted scripting environment.

  6. Once the buildout has finished, we can start Zope in debug mode with:
    $ bin/instance fg

When we see the line Zope is ready to handle requests, we can open a web browser and go to http://localhost:8080. We should see a welcome page, inviting you to create a new Plone site. Using the administrative username and password (admin/admin) specified in the buildout.cfg file, we can do so. For now, we will leave all options at their default settings.

Presuming we kept Plone as the Plone site id, we should now be able to access our Plone site at http://localhost:8080/Plone.

When we are done, we can shut down Zope by pressing Ctrl+C in the terminal window.

Buildout basics

The simple buildout (mentioned in the previous section) illustrates most of Buildout's core features. Let us look at these in a bit more detail.

The bootstrap.py script installs zc.buildout itself, and gives us the bin/buildout command. This looks for build instructions in a file called buildout.cfg in the current directory. An alternative file may be specified with the -c command line option:

$ bin/buildout -c somefile.cfg

Note

The bin/buildout command must be re-run for any changes to the buildout configuration file to take effect.

To run a buildout in offline mode, we can use:

$ bin/buildout -o

This is useful if we are not connected to the internet, or simply to speed up buildout execution.

Note

An offline build will fail unless all required distributions and files are available on the local machine. This usually means that you need to run the buildout at least once already.

See the output of bin/buildout --help for other options.

The buildout configuration file consists of sections, in square brackets, containing options, given as name = value pairs. Options that accept multiple values use whitespace as a delimiter. Such values may also be given on multiple, indented lines:

[instance]
eggs =
    Plone
    Products.PloneFormGen

A value can be referenced from another value as ${section:option}. For example:

[hosts]
devserver = localhost

[ports]
devserver = 8080

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
eggs = Plone
http-address = ${hosts:devserver}:${ports:devserver}

This kind of variable substitution becomes important once we start to define sections that are shared by multiple buildouts.

It is also possible to add to or remove from a multi-line option using += and -=:

[instance]
eggs =
    Plone
    Products.PloneFormGen

# later, possibly in another file:

[instance]
eggs +=
    collective.googleanalytics

Lines beginning with a # are taken as comments and ignored.

In a buildout configuration file, the [buildout] section, which controls global options, usually comes first:

[buildout]
extends =
    http://dist.plone.org/release/4.1/versions.cfg
parts = instance

The extends line is used to include other configuration files, which can be specified by relative path or remote URL. All included files are merged into a single logical buildout. If, for a particular named section, an option in an extended file is also specified in the extending file, the latter's value is used.

The extends line references a known good versions set for Plone. If you open it in a web browser, you will see something like this (truncated for brevity):

[buildout]
extends =
        http://download.zope.org/zopetoolkit/index/1.0.3/zopeapp-versions.cfg
        http://download.zope.org/Zope2/index/2.13.8/versions.cfg

[versions]
Plone                                 = 4.1
Products.ATContentTypes   = 2.1.3

This file extends two other files, which in turn contain statements like (again truncated):

[buildout]
versions = versions

[versions]
Zope2 = 2.13.8
...

When these files are merged, Buildout will look for version pins in the [versions] section (as indicated by the ${buildout:versions} option). The Zope 2 has known good set pins numerous distributions in its [versions] block, to which the Plone known good set adds its own. We could also add or override some version pins using a [versions] section in our own top level buildout.cfg.

The parts option in the [buildout] section is used to list, in order, the steps that Buildout should follow when executing the build. Each part is defined by a section containing a recipe option.

A recipe is a named distribution which exposes the logic to perform a particular build task. It will be downloaded from PyPI as required. Most recipes accept and/or require various options, read from the relevant part section. It is possible to use a given recipe multiple times (in different parts) in a single buildout.

Note

Writing a new recipe is beyond the scope of this book, but it is not particularly difficult. See http://buildout.org for details. With numerous recipes available on PyPI, however, custom recipes are rarely required. You can look for buildout recipes on PyPI: http://pypi.python.org/pypi?:action=browse&c=512

In the example above, we specified a single part, instance, which was defined in the following section:

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
eggs = Plone

This particular recipe is used to configure a Zope 2 instance, into which we install the Plone distribution, and configure an initial user called admin, with the password admin. See http://pypi.python.org/pypi/plone.recipe.zope2instance for details about other options supported by this recipe.

Several recipes, including plone.recipe.zope2instance, use the eggs option to define a working set of eggs that should be provided for any console scripts generated. When Buildout generates a script, it will embed the working set in the sys.path variable. For example, bin/instance will look something like this (truncated for brevity):

#!/usr/local/bin/python2.6

import sys
sys.path[0:0] = [
  '/path/to/buildout/eggs/Plone-4.1-py2.6.egg',
  '/path/to/buildout/eggs/bpython-0.9.7.1-py2.6.egg',
  '/path/to/buildout/eggs/plone.reload-1.5-py2.6.egg',
  ...
  ]

import plone.recipe.zope2instance.ctl

if __name__ == '__main__':
    plone.recipe.zope2instance.ctl.main(
        ["-C", '/path/to/buildout/parts/instance/etc/zope.conf']
        + sys.argv[1:])

This 'path mangling' ensures that the relevant packages, at the required versions, are available at runtime, and provides for isolation between scripts and buildouts.

The working set is calculated from all distributions listed in the eggs option, in addition to their dependencies. When the relevant part is executed, Buildout will download and install distributions from PyPI as necessary. Downloaded source distributions are kept in the dist directory in the downloads cache, if one is configured. Platform-specific egg installations are kept in the eggs/ directory, either locally in buildout root or in a shared eggs cache.

Alternatively, we can specify one or more develop eggs. If no version pin says otherwise, a develop egg normally takes precedence over downloaded distributions. During development, we will usually manage all our custom code as develop eggs in the src/ directory inside the buildout. Develop eggs must be explicitly listed in the develop option in the 'buildout' section. For example:

[buildout]
develop =
    src/my.package
    src/my.otherpackage

It is also possible to use a wildcard match:

[buildout]
develop =
    src/*

We will demonstrate a more comprehensive buildout. For more details about Buildout, see http://www.buildout.org/ and http://pypi.python.org/pypi/zc.buildout.

The buildout directory

Besides the buildout configuration and bootstrap files, a buildout consists of a number of directories and files, many of which are created and managed by Buildout and various recipes.

We always should put our build under source control, using a version control system such as Subversion or Git. As a rule of thumb, however, we should not version control any files generated by buildout. This usually means adding the relevant files or directories to the 'ignore' list for our source code management system.

Inside a Plone development buildout, we may find the following files and directories:

File or directory

Version control

Purpose

bootstrap.py

Yes

Installs zc.buildout in a freshly created buildout.

*.cfg

Yes

Buildout configuration files, including the default buildout.cfg.

.installed.cfg

No

Describes the currently installed buildout configuration. This allows buildout to run already-installed recipes in 'update', rather than 'install' mode.

.mr.developer.cfg

No

Describes currently installed develop eggs when using the mr.developer extension.

src/

Yes

The default location for custom distributions. You should version control the src/ directory itself, but if you use the mr.developer extension to manage your develop eggs, you should ignore src/* (all files inside the src/ directory) to avoid accidentally checking them into the build itself. See further for more details.

bin/

No

Contains installed scripts, such as bin/buildout and bin/instance.

eggs/

No

Contains all eggs ever installed from binary or source distributions. (The currently active set of eggs for a given runtime is embedded in the scripts in the bin/ directory.) It is possible to share an eggs directory among multiple buildouts.

develop-eggs/

No

Contains the placeholder files for currently active develop eggs.

parts/

No

Used by recipes to store internal files. Buildout is liable to delete or overwrite anything inside this directory at any time.

var/

No

Contains runtime configuration such as logs (var/logs/) and the Zope database files (var/filestorage/ and var/blobstorage/).

coverage/

No

Default output location for HTML test coverage reports created with z3c.coverage. We will describe this tool in more detail shortly.

*.pyc, *.pyo

No

These are byte-compiled Python files, which should never be under version control. They may appear in distributions inside the src/ directory, for example.

Tip

If all goes wrong and you want to perform a hard reset on your buildout, delete the parts/ directory and the .installed.cfg file, and rerun buildout. If you think some installed eggs may have been corrupted, you can delete the eggs/ and develop-eggs/ directories as well, which will cause distributions to be reinstalled.

Buildout defaults

To save bandwidth, disk space, and build time, it is a good idea to share third party files between buildouts. Buildout allows us to specify shared directories for eggs, downloads, and remotely extended buildout files (such as known good version sets).

Buildout defaults are set in a file found in ~/.buildout/default.cfg, where ~ is our home directory.

Tip

On Windows, you can find your home directory by opening the Python interpreter and running import os.path; print os.path.expanduser("~") at the interactive prompt.

We must first create the .buildout directory inside our home directory if it does not already exist. Inside .buildout, we then create the directories eggs, downloads, and extends to store the various types of downloads, and a file called default.cfg, containing:

[buildout]
eggs-directory = /path/to/.buildout/eggs
download-cache = /path/to/.buildout/downloads
extends-cache  = /path/to/.buildout/extends

Note

Be sure to update the paths to reflect the actual location of the .buildout directory on your system.

The default.cfg file can be used for other defaults if necessary. It behaves as if it is implicitly extended by any buildout you run.

Packages and distributions

Distributions using Setuptools/Distribute – include those we will write starting from Chapter 5, Developing a Site Strategy and use as develop eggs—consisting of a directory with a top-level setup.py file, and relevant source code, documentation, and other files.

setup.py contains metadata about the distribution itself, and declares its current version as well as any dependencies. Dependencies can be specified down to particular versions (such as ">=0.2,<1.0" means "later than version 0.2 but earlier than version 1.0"). Here is a sample setup.py file:

from setuptools import setup, find_packages
import os

version = '1.0'

setup(name='my.package',
      version=version,
      description="Description of my package",
      long_description=open("README.txt").read() + "\n" +
                   open(os.path.join("docs", "HISTORY.txt")).read(),
      author='Martin Aspeli',
      author_email='optilude@gmail.com',
      license='GPL',
      packages=find_packages(exclude=['ez_setup']),
      namespace_packages=['my'],
      include_package_data=True,
      zip_safe=False,
      install_requires=[
          'setuptools',
          'some.package >= 1.0',
      ],
      )

Here:

  • The distribution name is my.package, and the version is 1.0.
  • We have specified a short and long description, author information, and other metadata for PyPI. The long description is read from the files README.txt and docs/HISTORY.txt.
  • We ask Setuptools/Distribute to find source code in the current directory, excluding the module ez_setup if found. By default, this is done by looking for any files under version control.
  • We declare that my.* is a namespace package.
  • We declare that the distribution cannot be run from a zipped archive, as this is unsupported by Zope and discouraged in general.
  • We declare that this distribution depends on setuptools (to support namespace packages) and some.package. The latter has to be version 1.0 or later.

When a distribution is installed, Setuptools/Distribute will attempt to fulfill dependencies listed under install_requires by downloading and installing them if necessary. It will look for distributions on PyPI by default, but an alternative index can be specified if necessary, and secondary 'find-links' may be listed in the distribution itself.

It is possible to use setup.py to install a distribution into the global site packages of a given Python interpreter:

$ python setup.py install

Note

Do not do this for any Plone distributions. Use Buildout instead.

To install a develop egg instead, we can run:

$ python setup.py develop

This creates a placeholder file that links to the distribution's source code, allowing it to be added to the Python path at runtime.

New packages can be released as source distributions, which are just zip files of the package with some additional metadata. We can build a source distribution of a package with:

$ python setup.py sdist

The new distribution will be placed in the dist subdirectory, which will be created if necessary.

Distributions can be uploaded to PyPI through a setup.py command:

$ python setup.py egg_info -RDb "" sdist register upload

Note

On Mac OS X, you should make sure resource forks are not included in the archive, as they can confuse other operating systems. To do that, you can run two commands before the line above: export COPY_EXTENDED_ATTRIBUTES_DISABLE=true and export COPYFILE_DISABLE=true.

You will be asked to specify or create a PyPI account if this is the first time you run this command. The login details are stored in the .pypirc file in your home directory.

The easy_install script searches PyPI or a similar index for distribution to download and install into the global Python environment. The latest valid versions of any dependencies will be included automatically. For example, we installed Virtualenv with:

$ easy_install -U virtualenv

Some distributions have optional dependencies known as extras. These can be installed using square bracket notation, for example:

$ easy_install -U my.package [someextra]

The same notation is supported in the install_requires line for dependencies. For example, a distribution that wanted to declare an extra called test that added a dependency on plone.testing with the z2 extra could add the following in its setup.py file:

      extras_require={
          'test': ['plone.testing[z2]',]
      },

Distributions can contain metadata about included plugins using a mechanism called entry points. This allows code to discover these plugins at runtime. Entry points are listed in setup.py, under the entry_points argument. For example, some Plone add-ons use an entry point like this to automatically register their configuration with Plone:

      entry_points="""
      [z3c.autoinclude.plugin]
      target = plone
      """,

Note

For more information about Setuptools/Distribute and setup.py, see http://guide.python-distribute.org/.

The buildout directory

Besides the buildout configuration and bootstrap files, a buildout consists of a number of directories and files, many of which are created and managed by Buildout and various recipes.

We always should put our build under source control, using a version control system such as Subversion or Git. As a rule of thumb, however, we should not version control any files generated by buildout. This usually means adding the relevant files or directories to the 'ignore' list for our source code management system.

Inside a Plone development buildout, we may find the following files and directories:

File or directory

Version control

Purpose

bootstrap.py

Yes

Installs zc.buildout in a freshly created buildout.

*.cfg

Yes

Buildout configuration files, including the default buildout.cfg.

.installed.cfg

No

Describes the currently installed buildout configuration. This allows buildout to run already-installed recipes in 'update', rather than 'install' mode.

.mr.developer.cfg

No

Describes currently installed develop eggs when using the mr.developer extension.

src/

Yes

The default location for custom distributions. You should version control the src/ directory itself, but if you use the mr.developer extension to manage your develop eggs, you should ignore src/* (all files inside the src/ directory) to avoid accidentally checking them into the build itself. See further for more details.

bin/

No

Contains installed scripts, such as bin/buildout and bin/instance.

eggs/

No

Contains all eggs ever installed from binary or source distributions. (The currently active set of eggs for a given runtime is embedded in the scripts in the bin/ directory.) It is possible to share an eggs directory among multiple buildouts.

develop-eggs/

No

Contains the placeholder files for currently active develop eggs.

parts/

No

Used by recipes to store internal files. Buildout is liable to delete or overwrite anything inside this directory at any time.

var/

No

Contains runtime configuration such as logs (var/logs/) and the Zope database files (var/filestorage/ and var/blobstorage/).

coverage/

No

Default output location for HTML test coverage reports created with z3c.coverage. We will describe this tool in more detail shortly.

*.pyc, *.pyo

No

These are byte-compiled Python files, which should never be under version control. They may appear in distributions inside the src/ directory, for example.

Tip

If all goes wrong and you want to perform a hard reset on your buildout, delete the parts/ directory and the .installed.cfg file, and rerun buildout. If you think some installed eggs may have been corrupted, you can delete the eggs/ and develop-eggs/ directories as well, which will cause distributions to be reinstalled.

Buildout defaults

To save bandwidth, disk space, and build time, it is a good idea to share third party files between buildouts. Buildout allows us to specify shared directories for eggs, downloads, and remotely extended buildout files (such as known good version sets).

Buildout defaults are set in a file found in ~/.buildout/default.cfg, where ~ is our home directory.

Tip

On Windows, you can find your home directory by opening the Python interpreter and running import os.path; print os.path.expanduser("~") at the interactive prompt.

We must first create the .buildout directory inside our home directory if it does not already exist. Inside .buildout, we then create the directories eggs, downloads, and extends to store the various types of downloads, and a file called default.cfg, containing:

[buildout]
eggs-directory = /path/to/.buildout/eggs
download-cache = /path/to/.buildout/downloads
extends-cache  = /path/to/.buildout/extends

Note

Be sure to update the paths to reflect the actual location of the .buildout directory on your system.

The default.cfg file can be used for other defaults if necessary. It behaves as if it is implicitly extended by any buildout you run.

Packages and distributions

Distributions using Setuptools/Distribute – include those we will write starting from Chapter 5, Developing a Site Strategy and use as develop eggs—consisting of a directory with a top-level setup.py file, and relevant source code, documentation, and other files.

setup.py contains metadata about the distribution itself, and declares its current version as well as any dependencies. Dependencies can be specified down to particular versions (such as ">=0.2,<1.0" means "later than version 0.2 but earlier than version 1.0"). Here is a sample setup.py file:

from setuptools import setup, find_packages
import os

version = '1.0'

setup(name='my.package',
      version=version,
      description="Description of my package",
      long_description=open("README.txt").read() + "\n" +
                   open(os.path.join("docs", "HISTORY.txt")).read(),
      author='Martin Aspeli',
      author_email='optilude@gmail.com',
      license='GPL',
      packages=find_packages(exclude=['ez_setup']),
      namespace_packages=['my'],
      include_package_data=True,
      zip_safe=False,
      install_requires=[
          'setuptools',
          'some.package >= 1.0',
      ],
      )

Here:

  • The distribution name is my.package, and the version is 1.0.
  • We have specified a short and long description, author information, and other metadata for PyPI. The long description is read from the files README.txt and docs/HISTORY.txt.
  • We ask Setuptools/Distribute to find source code in the current directory, excluding the module ez_setup if found. By default, this is done by looking for any files under version control.
  • We declare that my.* is a namespace package.
  • We declare that the distribution cannot be run from a zipped archive, as this is unsupported by Zope and discouraged in general.
  • We declare that this distribution depends on setuptools (to support namespace packages) and some.package. The latter has to be version 1.0 or later.

When a distribution is installed, Setuptools/Distribute will attempt to fulfill dependencies listed under install_requires by downloading and installing them if necessary. It will look for distributions on PyPI by default, but an alternative index can be specified if necessary, and secondary 'find-links' may be listed in the distribution itself.

It is possible to use setup.py to install a distribution into the global site packages of a given Python interpreter:

$ python setup.py install

Note

Do not do this for any Plone distributions. Use Buildout instead.

To install a develop egg instead, we can run:

$ python setup.py develop

This creates a placeholder file that links to the distribution's source code, allowing it to be added to the Python path at runtime.

New packages can be released as source distributions, which are just zip files of the package with some additional metadata. We can build a source distribution of a package with:

$ python setup.py sdist

The new distribution will be placed in the dist subdirectory, which will be created if necessary.

Distributions can be uploaded to PyPI through a setup.py command:

$ python setup.py egg_info -RDb "" sdist register upload

Note

On Mac OS X, you should make sure resource forks are not included in the archive, as they can confuse other operating systems. To do that, you can run two commands before the line above: export COPY_EXTENDED_ATTRIBUTES_DISABLE=true and export COPYFILE_DISABLE=true.

You will be asked to specify or create a PyPI account if this is the first time you run this command. The login details are stored in the .pypirc file in your home directory.

The easy_install script searches PyPI or a similar index for distribution to download and install into the global Python environment. The latest valid versions of any dependencies will be included automatically. For example, we installed Virtualenv with:

$ easy_install -U virtualenv

Some distributions have optional dependencies known as extras. These can be installed using square bracket notation, for example:

$ easy_install -U my.package [someextra]

The same notation is supported in the install_requires line for dependencies. For example, a distribution that wanted to declare an extra called test that added a dependency on plone.testing with the z2 extra could add the following in its setup.py file:

      extras_require={
          'test': ['plone.testing[z2]',]
      },

Distributions can contain metadata about included plugins using a mechanism called entry points. This allows code to discover these plugins at runtime. Entry points are listed in setup.py, under the entry_points argument. For example, some Plone add-ons use an entry point like this to automatically register their configuration with Plone:

      entry_points="""
      [z3c.autoinclude.plugin]
      target = plone
      """,

Note

For more information about Setuptools/Distribute and setup.py, see http://guide.python-distribute.org/.

Buildout defaults

To save bandwidth, disk space, and build time, it is a good idea to share third party files between buildouts. Buildout allows us to specify shared directories for eggs, downloads, and remotely extended buildout files (such as known good version sets).

Buildout defaults are set in a file found in ~/.buildout/default.cfg, where ~ is our home directory.

Tip

On Windows, you can find your home directory by opening the Python interpreter and running import os.path; print os.path.expanduser("~") at the interactive prompt.

We must first create the .buildout directory inside our home directory if it does not already exist. Inside .buildout, we then create the directories eggs, downloads, and extends to store the various types of downloads, and a file called default.cfg, containing:

[buildout]
eggs-directory = /path/to/.buildout/eggs
download-cache = /path/to/.buildout/downloads
extends-cache  = /path/to/.buildout/extends

Note

Be sure to update the paths to reflect the actual location of the .buildout directory on your system.

The default.cfg file can be used for other defaults if necessary. It behaves as if it is implicitly extended by any buildout you run.

Packages and distributions

Distributions using Setuptools/Distribute – include those we will write starting from Chapter 5, Developing a Site Strategy and use as develop eggs—consisting of a directory with a top-level setup.py file, and relevant source code, documentation, and other files.

setup.py contains metadata about the distribution itself, and declares its current version as well as any dependencies. Dependencies can be specified down to particular versions (such as ">=0.2,<1.0" means "later than version 0.2 but earlier than version 1.0"). Here is a sample setup.py file:

from setuptools import setup, find_packages
import os

version = '1.0'

setup(name='my.package',
      version=version,
      description="Description of my package",
      long_description=open("README.txt").read() + "\n" +
                   open(os.path.join("docs", "HISTORY.txt")).read(),
      author='Martin Aspeli',
      author_email='optilude@gmail.com',
      license='GPL',
      packages=find_packages(exclude=['ez_setup']),
      namespace_packages=['my'],
      include_package_data=True,
      zip_safe=False,
      install_requires=[
          'setuptools',
          'some.package >= 1.0',
      ],
      )

Here:

  • The distribution name is my.package, and the version is 1.0.
  • We have specified a short and long description, author information, and other metadata for PyPI. The long description is read from the files README.txt and docs/HISTORY.txt.
  • We ask Setuptools/Distribute to find source code in the current directory, excluding the module ez_setup if found. By default, this is done by looking for any files under version control.
  • We declare that my.* is a namespace package.
  • We declare that the distribution cannot be run from a zipped archive, as this is unsupported by Zope and discouraged in general.
  • We declare that this distribution depends on setuptools (to support namespace packages) and some.package. The latter has to be version 1.0 or later.

When a distribution is installed, Setuptools/Distribute will attempt to fulfill dependencies listed under install_requires by downloading and installing them if necessary. It will look for distributions on PyPI by default, but an alternative index can be specified if necessary, and secondary 'find-links' may be listed in the distribution itself.

It is possible to use setup.py to install a distribution into the global site packages of a given Python interpreter:

$ python setup.py install

Note

Do not do this for any Plone distributions. Use Buildout instead.

To install a develop egg instead, we can run:

$ python setup.py develop

This creates a placeholder file that links to the distribution's source code, allowing it to be added to the Python path at runtime.

New packages can be released as source distributions, which are just zip files of the package with some additional metadata. We can build a source distribution of a package with:

$ python setup.py sdist

The new distribution will be placed in the dist subdirectory, which will be created if necessary.

Distributions can be uploaded to PyPI through a setup.py command:

$ python setup.py egg_info -RDb "" sdist register upload

Note

On Mac OS X, you should make sure resource forks are not included in the archive, as they can confuse other operating systems. To do that, you can run two commands before the line above: export COPY_EXTENDED_ATTRIBUTES_DISABLE=true and export COPYFILE_DISABLE=true.

You will be asked to specify or create a PyPI account if this is the first time you run this command. The login details are stored in the .pypirc file in your home directory.

The easy_install script searches PyPI or a similar index for distribution to download and install into the global Python environment. The latest valid versions of any dependencies will be included automatically. For example, we installed Virtualenv with:

$ easy_install -U virtualenv

Some distributions have optional dependencies known as extras. These can be installed using square bracket notation, for example:

$ easy_install -U my.package [someextra]

The same notation is supported in the install_requires line for dependencies. For example, a distribution that wanted to declare an extra called test that added a dependency on plone.testing with the z2 extra could add the following in its setup.py file:

      extras_require={
          'test': ['plone.testing[z2]',]
      },

Distributions can contain metadata about included plugins using a mechanism called entry points. This allows code to discover these plugins at runtime. Entry points are listed in setup.py, under the entry_points argument. For example, some Plone add-ons use an entry point like this to automatically register their configuration with Plone:

      entry_points="""
      [z3c.autoinclude.plugin]
      target = plone
      """,

Note

For more information about Setuptools/Distribute and setup.py, see http://guide.python-distribute.org/.

Packages and distributions

Distributions using Setuptools/Distribute – include those we will write starting from Chapter 5, Developing a Site Strategy and use as develop eggs—consisting of a directory with a top-level setup.py file, and relevant source code, documentation, and other files.

setup.py contains metadata about the distribution itself, and declares its current version as well as any dependencies. Dependencies can be specified down to particular versions (such as ">=0.2,<1.0" means "later than version 0.2 but earlier than version 1.0"). Here is a sample setup.py file:

from setuptools import setup, find_packages
import os

version = '1.0'

setup(name='my.package',
      version=version,
      description="Description of my package",
      long_description=open("README.txt").read() + "\n" +
                   open(os.path.join("docs", "HISTORY.txt")).read(),
      author='Martin Aspeli',
      author_email='optilude@gmail.com',
      license='GPL',
      packages=find_packages(exclude=['ez_setup']),
      namespace_packages=['my'],
      include_package_data=True,
      zip_safe=False,
      install_requires=[
          'setuptools',
          'some.package >= 1.0',
      ],
      )

Here:

  • The distribution name is my.package, and the version is 1.0.
  • We have specified a short and long description, author information, and other metadata for PyPI. The long description is read from the files README.txt and docs/HISTORY.txt.
  • We ask Setuptools/Distribute to find source code in the current directory, excluding the module ez_setup if found. By default, this is done by looking for any files under version control.
  • We declare that my.* is a namespace package.
  • We declare that the distribution cannot be run from a zipped archive, as this is unsupported by Zope and discouraged in general.
  • We declare that this distribution depends on setuptools (to support namespace packages) and some.package. The latter has to be version 1.0 or later.

When a distribution is installed, Setuptools/Distribute will attempt to fulfill dependencies listed under install_requires by downloading and installing them if necessary. It will look for distributions on PyPI by default, but an alternative index can be specified if necessary, and secondary 'find-links' may be listed in the distribution itself.

It is possible to use setup.py to install a distribution into the global site packages of a given Python interpreter:

$ python setup.py install

Note

Do not do this for any Plone distributions. Use Buildout instead.

To install a develop egg instead, we can run:

$ python setup.py develop

This creates a placeholder file that links to the distribution's source code, allowing it to be added to the Python path at runtime.

New packages can be released as source distributions, which are just zip files of the package with some additional metadata. We can build a source distribution of a package with:

$ python setup.py sdist

The new distribution will be placed in the dist subdirectory, which will be created if necessary.

Distributions can be uploaded to PyPI through a setup.py command:

$ python setup.py egg_info -RDb "" sdist register upload

Note

On Mac OS X, you should make sure resource forks are not included in the archive, as they can confuse other operating systems. To do that, you can run two commands before the line above: export COPY_EXTENDED_ATTRIBUTES_DISABLE=true and export COPYFILE_DISABLE=true.

You will be asked to specify or create a PyPI account if this is the first time you run this command. The login details are stored in the .pypirc file in your home directory.

The easy_install script searches PyPI or a similar index for distribution to download and install into the global Python environment. The latest valid versions of any dependencies will be included automatically. For example, we installed Virtualenv with:

$ easy_install -U virtualenv

Some distributions have optional dependencies known as extras. These can be installed using square bracket notation, for example:

$ easy_install -U my.package [someextra]

The same notation is supported in the install_requires line for dependencies. For example, a distribution that wanted to declare an extra called test that added a dependency on plone.testing with the z2 extra could add the following in its setup.py file:

      extras_require={
          'test': ['plone.testing[z2]',]
      },

Distributions can contain metadata about included plugins using a mechanism called entry points. This allows code to discover these plugins at runtime. Entry points are listed in setup.py, under the entry_points argument. For example, some Plone add-ons use an entry point like this to automatically register their configuration with Plone:

      entry_points="""
      [z3c.autoinclude.plugin]
      target = plone
      """,

Note

For more information about Setuptools/Distribute and setup.py, see http://guide.python-distribute.org/.

The development buildout

We will now create the development buildout that will be used throughout this book. You can find this in the book's accompanying source code.

We currently have three buildout configuration files , all found at the root of the buildout:

File

Purpose

versions.cfg

Contains our own known good set of distributions, allowing us to pin down custom dependencies and override any version pins from other known good sets we extend.

packages.cfg

Extends versions.cfg and other known good version sets, including the known good set for Plone, sets some global options, and defines various working sets. This file allows us to manage the distributions that make up our project in a way that can be reused by multiple top-level buildout configuration files.

buildout.cfg

This is the top-level buildout file for our development build, containing the parts and options we need to set up a development environment. Later in the book, we will add additional top-level buildout files for production deployment.

The first of these files, versions.cfg, is straightforward:

# Project-specific version pins
# =============================

[versions]
# Buildout
mr.developer = 1.17
collective.recipe.omelette = 0.10

# Development tools
bpython = 0.9.7.1
pygments = 1.4
Products.DocFinderTab = 1.0.4
Products.PDBDebugMode = 1.3.1
Products.PrintingMailHost = 0.7
z3c.coverage = 1.2.0
jarn.mkrelease = 3.0.9
setuptools-git = 0.4.2
setuptools-hg = 0.2

# ZopeSkel
ZopeSkel = 2.19
Cheetah = 2.2.1
Paste = 1.7.5.1
PasteScript = 1.7.3
PasteDeploy = 1.3.4

Note

The version pins shown here were the most appropriate versions at the time of writing. You may want to update some of these versions. See the description of z3c.checkversions mentioned in the following sections for details.

Next up, packages.cfg is a little more interesting:

# Information about packages and known good version sets
# ======================================================

[buildout]
extensions = mr.developer buildout.dumppickedversions
extends = 
# Known good sets of eggs we may be using
    http://dist.plone.org/release/4.1/versions.cfg
    versions.cfg

versions = versions
unzip = true

# Egg sets
[eggs]
main =
    Plone
test = 
devtools =
    bpython
    plone.reload
    Products.PDBDebugMode
    Products.PrintingMailHost
    Products.DocFinderTab

# Checkout locations
[sources]

This first installs two buildout extensions: mr.developer, which is used to manage develop eggs, and buildout.dumppickedversions, which will print the picked versions for any unpinned distributions at the end of the buildout run. We will describe these in more detail in the next section.

Next, we extend two known good sets—the one for our chosen Plone version, and our own local versions.cfg, which should come last so that it can override any versions from the external sets.

We also set the unzip option to true, which means that buildout will always unzip eggs in eggs/ directory. This makes debugging easier, avoids potential problems with Zope code that does not properly set the zip_safe flag to False, and is a prerequisite for using collective.recipe.omelette. Two recipe-less sections are then defined: [sources] is used by mr.developer, and will be described in a moment. [eggs] is used to define working sets that are referenced in other buildout sections:

Working set

Purpose

main

The minimal set of distributions required to run our application. Until we create some distributions of our own, we use the Plone egg here. In Chapter 5, Developing a Site Strategy we will replace this with our own 'policy package', which in turn will depend on all the software we need, including Plone.

test

This option lists all distributions for which we want to run automated tests. Usually, that means every custom distribution we create. If distributions have a [test] extra for test-only dependencies, this should be used here. Until we create our own distributions, this is empty.

devtools

Additional distributions that we want to install in the development environment, but not on a production server.

Finally, buildout.cfg contains the actual build instructions for our development environment:

# Development environment buildout
# ================================

[buildout]
parts =
    instance
    test
    coverage-report
    omelette
    zopepy
    zopeskel
    checkversions
    mkrelease

extends =
    packages.cfg

# Packages to check out/update when buildout is run
auto-checkout =
    
# Make sure buildout always attempts to update packages
always-checkout = force

# Development Zope instance. Installs the ``bin/instance`` script
[instance]
recipe = plone.recipe.zope2instance
http-address = 8080
user = admin:admin
verbose-security = on
eggs =
    ${eggs:main}
    ${eggs:devtools}
# Test runner. Run: ``bin/test`` to execute all tests
[test]
recipe = zc.recipe.testrunner
eggs = ${eggs:test}
defaults = ['--auto-color', '--auto-progress']

# Coverage report generator.
# Run: ``bin/test --coverage=coverage``
# and then: ``bin/coveragereport``
[coverage-report]
recipe = zc.recipe.egg
eggs = z3c.coverage
scripts = coveragereport
arguments = ('parts/test/coverage', 'coverage')

# Installs links to all installed packages to ``parts/omelette``.
# On Windows, you need to install junction.exe first
[omelette]
recipe = collective.recipe.omelette
eggs = 
    ${eggs:main}
    ${eggs:devtools}

# Installs the ``bin/zopepy`` interpreter.
[zopepy]
recipe = zc.recipe.egg
eggs = 
    ${eggs:main}
    ${eggs:devtools}
interpreter = zopepy

# Installs ZopeSkel, which can be used to create new packages
# Run: ``bin/zopeskel``
[zopeskel]
recipe = zc.recipe.egg
eggs = ZopeSkel

# Tool to help check for new versions.
# Run: ``bin/checkversions versions.cfg``
[checkversions]
recipe = zc.recipe.egg
eggs = z3c.checkversions [buildout]

# Tool to make releases
# Run: ``bin/mkrelease --help``
[mkrelease]
recipe = zc.recipe.egg
eggs = jarn.mkrelease

First, we list the parts to execute, and extend packages.cfg to gain access to our working- and known good version sets. The last two options in the [buildout] section, auto-checkout and always-checkout, are used by mr.developer and described below.

Next, we define our development Zope instance. This runs on port 8080, has an administrative Zope user called admin with password admin, provides additional debug information for security violations, and is installed with a working set combining ${eggs:main} and ${eggs:devtools}.

The subsequent parts install various development tools, which are described in more detail in the subsequent sections.

Note

You may have noticed that several parts use zc.recipe.egg. This recipe is used to install one or more distributions. If a distribution declares one or more console scripts (through an entry point), these are installed to the bin/ directory refer http://pypi.python.org/pypi/zc.recipe.egg for details.

Development tools

Let us now examine the various development tools that we will use throughout the book.

Buildout extensions

The first two development tools are extensions to buildout itself, which operate at build time.

mr.developer

We previously saw how to list develop eggs using the ${buildout:develop} option. As our projects grow, it is usually beneficial to version control the buildout separately from the distributions used as we develop eggs. This allows us to release, tag, and branch individual distributions independently of other distributions, and of the build itself. We can do this manually by checking out distributions to the src/ directory and keeping the develop option up to date, but this is cumbersome and error prone. The mr.developer extension was built to alleviate this. It can look for packages that may be used as develop eggs in the [sources] section, which we have placed in our packages.cfg file. For example:

[sources]
my.package = svn https://svn.example.org/repos/my.package/trunk
my.otherpackage = git https://github.com/myorg/my.otherpackage

Here, two distributions are configured for mr.developer, one found in a Subversion repository, and one found in a Git repository. (Other version control systems, including Mercurial, Bzr, and CVS are supported as well.) If a package is not (yet) checked in to any version control system, we can manually place it in the src/ directory and list it with the fs option (short for filesystem):

[sources]
optilux.policy = fs optilux.policy

Tip

Most Plone distributions are found in one of the standard Plone repositories: http://svn.plone.org/svn/plone for core distributions, or http://svn.plone.org/svn/collective for community add-ons. If you need to use an unreleased distribution (for example to get access to a recent bug fix), you can add it to mr.developer's sources list with the appropriate Subversion URL as well.

With [sources] defined, we can tell mr.developer to automatically check out distributions and register them as develop eggs using the ${buildout:auto-checkout} option. For development purposes, we have placed this in our top-level buildout.cfg file. To use the two fictitious distributions in our previous example as develop eggs, we could do:

[buildout]
...
auto-checkout =
    my.package
    my.otherpackage

Note

We do this in buildout.cfg, and not packages.cfg, because we only want to track the source code repository during development. In the final part of this book, we will describe how to make internal distribution releases suitable for production deployment.

If we also want to ensure that distributions are updated from the source code repository each time buildout is run, we can add always-checkout = true to the [buildout] section. By default, packages are checked out once, but must be manually updated.

Note

There is always a chance that an update could result in a merge conflict if uncommitted local changes interfere with changes pulled from a remote repository. In this case, mr.developer will abort the buildout, allowing you to resolve the merge conflict before running the build again.

mr.developer also installs a script called bin/develop. This can be used to update one or all distributions tracked by mr.developer, activate or deactivate develop eggs, track the status of all distributions (even across multiple version control systems and repositories), and rerun buildout (which is required for distributions activations/deactivations to take effect). See the output of bin/develop help for details.

This covers the most basic use of mr.developer. See http://pypi.python.org/pypi/mr.developer for further options and usage patterns.

buildout.dumppickedversions

As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.

The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:

*************** PICKED VERSIONS ****************
[versions]
Products.PDBDebugMode = 1.1
Products.PrintingMailHost = 0.7
*************** /PICKED VERSIONS ***************

Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.

Tip

The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.

Development Zope instance

For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.

Note

'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.

During development, we normally run Zope in foreground mode, with:

$ bin/instance fg

Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.

Note

Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.

We also install various development tools into the Zope instance. These are outlined in the subsequent sections.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Development tools

Let us now examine the various development tools that we will use throughout the book.

Buildout extensions

The first two development tools are extensions to buildout itself, which operate at build time.

mr.developer

We previously saw how to list develop eggs using the ${buildout:develop} option. As our projects grow, it is usually beneficial to version control the buildout separately from the distributions used as we develop eggs. This allows us to release, tag, and branch individual distributions independently of other distributions, and of the build itself. We can do this manually by checking out distributions to the src/ directory and keeping the develop option up to date, but this is cumbersome and error prone. The mr.developer extension was built to alleviate this. It can look for packages that may be used as develop eggs in the [sources] section, which we have placed in our packages.cfg file. For example:

[sources]
my.package = svn https://svn.example.org/repos/my.package/trunk
my.otherpackage = git https://github.com/myorg/my.otherpackage

Here, two distributions are configured for mr.developer, one found in a Subversion repository, and one found in a Git repository. (Other version control systems, including Mercurial, Bzr, and CVS are supported as well.) If a package is not (yet) checked in to any version control system, we can manually place it in the src/ directory and list it with the fs option (short for filesystem):

[sources]
optilux.policy = fs optilux.policy

Tip

Most Plone distributions are found in one of the standard Plone repositories: http://svn.plone.org/svn/plone for core distributions, or http://svn.plone.org/svn/collective for community add-ons. If you need to use an unreleased distribution (for example to get access to a recent bug fix), you can add it to mr.developer's sources list with the appropriate Subversion URL as well.

With [sources] defined, we can tell mr.developer to automatically check out distributions and register them as develop eggs using the ${buildout:auto-checkout} option. For development purposes, we have placed this in our top-level buildout.cfg file. To use the two fictitious distributions in our previous example as develop eggs, we could do:

[buildout]
...
auto-checkout =
    my.package
    my.otherpackage

Note

We do this in buildout.cfg, and not packages.cfg, because we only want to track the source code repository during development. In the final part of this book, we will describe how to make internal distribution releases suitable for production deployment.

If we also want to ensure that distributions are updated from the source code repository each time buildout is run, we can add always-checkout = true to the [buildout] section. By default, packages are checked out once, but must be manually updated.

Note

There is always a chance that an update could result in a merge conflict if uncommitted local changes interfere with changes pulled from a remote repository. In this case, mr.developer will abort the buildout, allowing you to resolve the merge conflict before running the build again.

mr.developer also installs a script called bin/develop. This can be used to update one or all distributions tracked by mr.developer, activate or deactivate develop eggs, track the status of all distributions (even across multiple version control systems and repositories), and rerun buildout (which is required for distributions activations/deactivations to take effect). See the output of bin/develop help for details.

This covers the most basic use of mr.developer. See http://pypi.python.org/pypi/mr.developer for further options and usage patterns.

buildout.dumppickedversions

As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.

The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:

*************** PICKED VERSIONS ****************
[versions]
Products.PDBDebugMode = 1.1
Products.PrintingMailHost = 0.7
*************** /PICKED VERSIONS ***************

Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.

Tip

The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.

Development Zope instance

For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.

Note

'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.

During development, we normally run Zope in foreground mode, with:

$ bin/instance fg

Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.

Note

Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.

We also install various development tools into the Zope instance. These are outlined in the subsequent sections.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Buildout extensions

The first two development tools are extensions to buildout itself, which operate at build time.

mr.developer

We previously saw how to list develop eggs using the ${buildout:develop} option. As our projects grow, it is usually beneficial to version control the buildout separately from the distributions used as we develop eggs. This allows us to release, tag, and branch individual distributions independently of other distributions, and of the build itself. We can do this manually by checking out distributions to the src/ directory and keeping the develop option up to date, but this is cumbersome and error prone. The mr.developer extension was built to alleviate this. It can look for packages that may be used as develop eggs in the [sources] section, which we have placed in our packages.cfg file. For example:

[sources]
my.package = svn https://svn.example.org/repos/my.package/trunk
my.otherpackage = git https://github.com/myorg/my.otherpackage

Here, two distributions are configured for mr.developer, one found in a Subversion repository, and one found in a Git repository. (Other version control systems, including Mercurial, Bzr, and CVS are supported as well.) If a package is not (yet) checked in to any version control system, we can manually place it in the src/ directory and list it with the fs option (short for filesystem):

[sources]
optilux.policy = fs optilux.policy

Tip

Most Plone distributions are found in one of the standard Plone repositories: http://svn.plone.org/svn/plone for core distributions, or http://svn.plone.org/svn/collective for community add-ons. If you need to use an unreleased distribution (for example to get access to a recent bug fix), you can add it to mr.developer's sources list with the appropriate Subversion URL as well.

With [sources] defined, we can tell mr.developer to automatically check out distributions and register them as develop eggs using the ${buildout:auto-checkout} option. For development purposes, we have placed this in our top-level buildout.cfg file. To use the two fictitious distributions in our previous example as develop eggs, we could do:

[buildout]
...
auto-checkout =
    my.package
    my.otherpackage

Note

We do this in buildout.cfg, and not packages.cfg, because we only want to track the source code repository during development. In the final part of this book, we will describe how to make internal distribution releases suitable for production deployment.

If we also want to ensure that distributions are updated from the source code repository each time buildout is run, we can add always-checkout = true to the [buildout] section. By default, packages are checked out once, but must be manually updated.

Note

There is always a chance that an update could result in a merge conflict if uncommitted local changes interfere with changes pulled from a remote repository. In this case, mr.developer will abort the buildout, allowing you to resolve the merge conflict before running the build again.

mr.developer also installs a script called bin/develop. This can be used to update one or all distributions tracked by mr.developer, activate or deactivate develop eggs, track the status of all distributions (even across multiple version control systems and repositories), and rerun buildout (which is required for distributions activations/deactivations to take effect). See the output of bin/develop help for details.

This covers the most basic use of mr.developer. See http://pypi.python.org/pypi/mr.developer for further options and usage patterns.

buildout.dumppickedversions

As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.

The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:

*************** PICKED VERSIONS ****************
[versions]
Products.PDBDebugMode = 1.1
Products.PrintingMailHost = 0.7
*************** /PICKED VERSIONS ***************

Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.

Tip

The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.

Development Zope instance

For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.

Note

'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.

During development, we normally run Zope in foreground mode, with:

$ bin/instance fg

Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.

Note

Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.

We also install various development tools into the Zope instance. These are outlined in the subsequent sections.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

mr.developer

We previously saw how to list develop eggs using the ${buildout:develop} option. As our projects grow, it is usually beneficial to version control the buildout separately from the distributions used as we develop eggs. This allows us to release, tag, and branch individual distributions independently of other distributions, and of the build itself. We can do this manually by checking out distributions to the src/ directory and keeping the develop option up to date, but this is cumbersome and error prone. The mr.developer extension was built to alleviate this. It can look for packages that may be used as develop eggs in the [sources] section, which we have placed in our packages.cfg file. For example:

[sources]
my.package = svn https://svn.example.org/repos/my.package/trunk
my.otherpackage = git https://github.com/myorg/my.otherpackage

Here, two distributions are configured for mr.developer, one found in a Subversion repository, and one found in a Git repository. (Other version control systems, including Mercurial, Bzr, and CVS are supported as well.) If a package is not (yet) checked in to any version control system, we can manually place it in the src/ directory and list it with the fs option (short for filesystem):

[sources]
optilux.policy = fs optilux.policy

Tip

Most Plone distributions are found in one of the standard Plone repositories: http://svn.plone.org/svn/plone for core distributions, or http://svn.plone.org/svn/collective for community add-ons. If you need to use an unreleased distribution (for example to get access to a recent bug fix), you can add it to mr.developer's sources list with the appropriate Subversion URL as well.

With [sources] defined, we can tell mr.developer to automatically check out distributions and register them as develop eggs using the ${buildout:auto-checkout} option. For development purposes, we have placed this in our top-level buildout.cfg file. To use the two fictitious distributions in our previous example as develop eggs, we could do:

[buildout]
...
auto-checkout =
    my.package
    my.otherpackage

Note

We do this in buildout.cfg, and not packages.cfg, because we only want to track the source code repository during development. In the final part of this book, we will describe how to make internal distribution releases suitable for production deployment.

If we also want to ensure that distributions are updated from the source code repository each time buildout is run, we can add always-checkout = true to the [buildout] section. By default, packages are checked out once, but must be manually updated.

Note

There is always a chance that an update could result in a merge conflict if uncommitted local changes interfere with changes pulled from a remote repository. In this case, mr.developer will abort the buildout, allowing you to resolve the merge conflict before running the build again.

mr.developer also installs a script called bin/develop. This can be used to update one or all distributions tracked by mr.developer, activate or deactivate develop eggs, track the status of all distributions (even across multiple version control systems and repositories), and rerun buildout (which is required for distributions activations/deactivations to take effect). See the output of bin/develop help for details.

This covers the most basic use of mr.developer. See http://pypi.python.org/pypi/mr.developer for further options and usage patterns.

buildout.dumppickedversions

As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.

The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:

*************** PICKED VERSIONS ****************
[versions]
Products.PDBDebugMode = 1.1
Products.PrintingMailHost = 0.7
*************** /PICKED VERSIONS ***************

Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.

Tip

The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.

Development Zope instance

For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.

Note

'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.

During development, we normally run Zope in foreground mode, with:

$ bin/instance fg

Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.

Note

Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.

We also install various development tools into the Zope instance. These are outlined in the subsequent sections.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

buildout.dumppickedversions

As previously discussed, it is important to pin down distribution versions as a project matures, to ensure a build can be safely repeated in the future without the risk of a new distribution uploaded to PyPI causing a build or runtime error. At the very least, Buildout should not be given free rein to pick any distribution versions when a project is released and deployed to a production server.

The buildout.dumppickedversions extension will print a summary of any versions that were picked by Buildout at the end of each buildout run. The output looks like this:

*************** PICKED VERSIONS ****************
[versions]
Products.PDBDebugMode = 1.1
Products.PrintingMailHost = 0.7
*************** /PICKED VERSIONS ***************

Here, we have neglected to pin Products.PdbDebugMode and Products.PrintingMailHost. Helpfully, buildout.dumppickedversions has listed them in a format suitable for copying into our versions.cfg file. Once we do that and rerun buildout, they should disappear from this list.

Tip

The picked versions block is output even if buildout encountered an error. This can sometimes make it harder to spot error messages. Be sure to look above the PICKED VERSIONS line to check that buildout completed successfully.

Development Zope instance

For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.

Note

'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.

During development, we normally run Zope in foreground mode, with:

$ bin/instance fg

Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.

Note

Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.

We also install various development tools into the Zope instance. These are outlined in the subsequent sections.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Development Zope instance

For development, we usually use a single Zope instance, controlled by the script bin/instance (the script name is taken from the part name in the buildout configuration), with a local Data.fs file and blobstorage directory containing the Zope database. This instance is optimized for use in a development environment, and has the verbose-security flag turned on, to give more details about security-related errors.

Note

'Permission denied' errors are normally not reported in the error logs. To see them, go to the Errors control panel under Plone's Site Setup and remove Unauthorized from the list of ignored exception types.

During development, we normally run Zope in foreground mode, with:

$ bin/instance fg

Not only does this make it easier to see log messages and use the debugger, it also ensures that Zope runs in debug mode, which, among other things, ensures that page templates on the filesystem can be modified and their changes take effect without restarting Zope, and disables caching and merging of Plone's CSS and JavaScript resources.

Note

Note that this renders the debug-mode option of plone.recipe.zope2instance obsolete.

We also install various development tools into the Zope instance. These are outlined in the subsequent sections.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

plone.reload

Ordinarily, any change to Python or ZCML files requires a restart of Zope to take effect. This can be time consuming, especially if we are making frequent changes. plone.reload attempts to alleviate this by detecting changed code and reloading it.

Note

plone.reload only works when Zope is running in debug mode.

The reloader is invoked from the @@reload view at the root of the Zope instance. If Zope is running on port 8080, that means going to http://localhost:8080/@@reload. Click Reload Code to reload any code changed since startup or the previous reload. Click Reload Code and ZCML to also reload ZCML component configuration.

Note

The reload mechanism works most of the time, but it is not perfect, and may fail to detect certain changes. In rare circumstances, a reload may also cause difficult-to-debug crashes. If you see any problems after a reload, or the reload did not appear to work, you should restart Zope before attempting any other debugging.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Products.PdbDebugMode

This package installs an exception hook that will drop us to a debugging prompt if any exceptions are raised, which is useful for diagnosing exceptions. See the following sections for more details about PDB, the Python debugger.

Tip

Zope will not return a response while at a breakpoint, which may make the site appear to hang. Be sure to check the terminal where Zope is running for a break point if your browser does not complete a request as expected. Press c and then Enter to exit the debugger. You may need to do this multiple times if multiple threads have reached the same break point simultaneously.

Products.PdbDebugMode also provides a view called @@pdb, which can be used to drop into a PDB prompt at will. This is useful for ad-hoc introspection, or just to test a Python expression against a live site. Simply append /@@pdb to the URL of any content item (or the Plone site root), such as, http://localhost:8080/Plone/@@pdb. Use self.context at this prompt to inspect the relevant content object.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Products.PrintingMailHost

This package hooks into the Plone MailHost object to make it print the output of mail messages to the console instead of sending them to a mail relay. This is useful for testing and debugging code that sends e-mails.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Products.DocFinderTab

This package adds a Doc tab to most objects in the Zope Management Interface. We can use this to look at the methods, variables, and base classes of a given content object.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Test runner

Throughout this book, we will be writing automated tests for our code. To run those tests, we need a test runner, which is installed with zc.recipe.testrunner.

Note

In Plone 3, we would normally use the command bin/instance test to run tests. This is no longer supported in Zope 2.12, and thus Plone 4.

This recipe takes an eggs option, under which we list every distribution we want to test. In our buildout, this is initialized from ${eggs:test} from packages.cfg.

Note

Note that each distribution containing packages to test must be listed explicitly, even if it is a dependency of another included distribution.

The test runner recipe generates a script called bin/test (the name is taken from the test runner part's section). To run all tests, simply do:

$ bin/test

To run only the tests in a particular package, use the -s option:

$ bin/test -s my.package

To run only a test with a name matching a particular regular expression, use the -t option:

$ bin/test -s my.package -t test_something

See the output of bin/test --help for other options.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Coverage reporting

Ideally, we should have automated tests covering every code path in our application. To help measure how good our code coverage is, we can use the test runner's coverage reporting tool. This is enabled with the --coverage option, which should indicate a directory for the coverage report. For example:

$ bin/test --coverage=coverage

Note

Coverage analysis significantly slows down the test runner, which is why this option is not enabled by default.

This will output summary statistics to the console, and place the raw coverage reports in the directory parts/test/coverage. To turn these into a user-friendly HTML report, we can use the bin/coveragereport script installed by the z3c.coverage package:

$ bin/coverage

This looks in the aforementioned directory and outputs a coverage report to the coverage/ directory inside the buildout root. Open coverage/all.html in a web browser to see the full report, including line-by-line test coverage analysis.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Continuous integration

When working in a team, it is important to run the build tests regularly, with instant notification when either the build itself or a test breaks: early detection makes regressions easy to diagnose and resolve, and is an important element of code quality management. This style of working is known as continuous integration.

Instructions for setting up a continuous integration server are beyond the scope of this book, but you are encouraged to look at Hudson, an open source continuous integration tool that is easy to install and use. Hudson can be configured to execute a series of shell commands, including bin/buildout and bin/test. Recent versions of the test runner also support Subunit output, which can be converted to JUnit-style XML reports suitable for Hudson. See http://hudson-ci.org/ and http://pypi.python.org/pypi/python-subunit for more information.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Omelette

It is useful to be able to search the 'active' source code for debugging and analysis purposes. With numerous distributions making up the Plone working set, however, it can be difficult to keep track of which packages are currently in use.

To make this easier, we can use collective.recipe.omelette. This presents a single 'virtual' source tree of the current working set by creating symbolic links to all packages in all (unzipped) eggs in the directory parts/omelette.

Note

collective.recipe.omelette works on Windows, but there are three important caveats:

  1. You must download junction.exe (see http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), copy it to a directory on the system PATH, and run it manually at least once to accept its license terms.
  2. junction.exe is fairly slow and can affect buildout performance, especially on slower systems.
  3. junction.exe creates 'hard links', which means that if the parts/omelette directory is deleted, the original files inside the relevant eggs will also disappear. collective.recipe.omelette handles 'delinking' properly when buildout is rerun, but if you manually delete the parts/ directory, you will also need to delete the eggs/ directory to force a re-download of all code.

If you prefer not to use collective.recipe.omelette on Windows, simply remove it from the parts list.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

The zopepy interpreter

Python lets us quickly prototype code on the interactive interpreter prompt. This is very powerful and can save much guessing. To be able to import Zope and Plone code, however, we need the relevant set of packages available. This is achieved with the bin/zopepy script, which is simply a Python interpreter with the correct sys.path mangling applied.

Note

Bear in mind that zopepy does not actually start Zope or load any of its configuration, so Zope runtime state like configured components or database connections will not be available. If you need that, you can run bin/instance debug instead, which will start up Zope and present an interactive prompt. Note that if you make changes to the ZODB, you will need to explicitly commit them with: import transaction; transaction.commit().

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

ZopeSkel

ZopeSkel is a collection of skeleton templates that can be used to create new distributions quickly. It is installed as bin/zopeskel. We will see how to use this in Chapter 5, Developing a Site Strategy, but in the meantime, you can run bin/zopeskel --help to learn how to use it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

z3c.checkversions

We have extolled the virtues of pinning distributions a few times already, but one downside of maintaining version pins is that discovering new releases requires manually checking PyPI or other sources. For major components with their own known good version sets such as Plone, this is not so much of an issue, but for our various development tools and third-party dependencies, it can become a hassle.

To help with this, we can use the bin/checkversions script installed by z3c.checkversions. To run it against our versions.cfg file, we would do:

$ bin/checkversions versions.cfg

This will report the latest available version on PyPI. If we are only interested in bug fixes, we can restrict the check to minor versions, with:

$ bin/checkversions versions.cfg --level=2

See the output of bin/checkversions --help for more options.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

jarn.mkrelease

Eventually, we will need to make releases of our distributions. For distributions we intend to release to the public that can mean uploading source distributions to PyPI. For internal distributions, we usually want to release to an internal distribution location, so that we have immutable, versioned distributions for deployment.

As we will show in the final part of this book, we can use bin/mkrelease, installed by jarn.mkrelease, to easily make releases. For more information, see http://pypi.python.org/pypi/jarn.mkrelease.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Tools in the browser

The humble web browser is an important part of our development environment. Not only does it allow us to manually test our site; modern browsers also come with development tools that allow us to debug JavaScript, understand what CSS is affecting what elements on the page, prototype JavaScript code, and CSS changes, inspect request and response headers, and inspect and modify the HTML structure on the fly.

Chrome and Safari both have built-in development tools. For Firefox, there is the ever popular Firebug. Get it from http://getfirebug.com, and wonder how you ever lived without it.

Learning to help yourself

During development, there will probably be times when you are stumped. Plone is fairly well-documented, but the documentation is certainly not perfect. The mailing lists and chat room are great resources if you need help, but it is also very important to learn how to help yourself.

Find the documentation

Documentation for Plone code can usually be found in one of the following places:

  • High level and tutorial-style documentation is normally found at http://plone.org/documentation.
  • Many distributions include documentation in their PyPI listings. For example, plone.testing's documentation can be found at http://pypi.python.org/pypi/plone.testing.
  • Distributions often include documentation in their README file. This is often also the source of the documentation published to PyPI.
  • Internal documentation in the form of comments or tests is often very valuable. Most packages use Zope interfaces (more on those in Chapter 9, Nine Core Concepts of Zope Programming) to describe their key APIs. Look for a file called interfaces.py.

Use the Source, Luke!

Python's readability is both a blessing and a curse. A blessing because it is normally possible to read the source code to find out what is going on. A curse because this sometimes makes developers a little lax about documentation.

One of the first hurdles new developers should overcome is any undue respect for the Python files that make up Zope and Plone. There is (almost) nothing magical about them. Earlier, we showed how to install an 'omelette' (in the parts/omelette directory inside the buildout root) that provides a single view of the source code that makes up the Plone runtime environment for a project. This is the best place to look for source code.

Get used to searching for code using grep or equivalent graphical tools, opening them and looking for specific classes and methods. Seeing what a piece of code does can often be faster than looking up documentation or examples. As time goes by, you will find that a few packages come up repeatedly, and finding code will be easier.

You can of course change these files as well. A backup is advisable, but if you think that temporarily raising an exception, adding a PDB break point, or printing a message from somewhere deep inside Zope helps you to solve a problem, go right ahead.

Note

It is a bad idea to make permanent changes this way, however, not least because those changes will be overwritten if you upgrade or reinstall the particular component. In the next chapter, we will learn more about other ways of customizing code Zope and Plone. However, if you find a bug, please report it (at http://dev.plone.org/plone), and attach a patch if you can!

Become familiar with the debugger

PDB, the Python debugger, is your best friend. To insert a breakpoint in your code or in some other code that you are trying to debug add the following line and (re-)start Zope in the foreground in a terminal:

import pdb; pdb.set_trace()

Tip

To avoid a restart, you can usually use the @@reload view from plone.reload. Refer the previous sections.

When this line is encountered, execution will stop, and the terminal will display:

(pdb)

This is the interactive PDB prompt. Type help and press Enter to see available commands. The most important ones are pp, to print a variable or the result of an expression; n to step to the next line; s to step into a function call; l to show a listing of the source code around the current execution point; tbreak to add a temporary breakpoint at a particular line, and c, to stop debugging and continue execution until another breakpoint is encountered.

If you want to quickly test syntax or libraries, you can run Python's interactive interpreter, through the bin/zopepy script as described earlier.

Look at the logs

During development, you should typically run Plone in a terminal window, using bin/instance fg. This enables debug mode and prints log messages to the console. If anything goes wrong, you should inspect the log first.

If an exception is encountered, you will likely see a traceback. For example:

Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.CMFPlone.FactoryTool, line 361, in __call__
Module Products.CMFPlone.FactoryTool, line 147, in __getitem__
Module Products.CMFPlone.PloneFolder, line 406, in invokeFactory
Module Products.CMFCore.TypesTool, line 934, in constructContent
Module Products.CMFCore.TypesTool, line 345, in constructInstance
Module Products.CMFCore.TypesTool, line 357, in _finishConstruction
Module Products.CMFCore.CMFCatalogAware, line 145, in notifyWorkflowCreated
Module Products.CMFCore.WorkflowTool, line 355, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 392, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 476, in _changeStateOf
Module Products.DCWorkflow.DCWorkflow, line 571, in _executeTransition
Module Products.DCWorkflow.DCWorkflow, line 435, in updateRoleMappingsFor
Module Products.DCWorkflow.utils, line 60, in modifyRolesForPermission
Module AccessControl.Permission, line 93, in setRoles
AttributeError: appname

The actual error is usually on the last line of the traceback. If this is not in any code that you wrote, chances are something you did further up the stack caused the problem, if you passed an invalid argument to a function. Start from the end and work your way up the stack trace until you see some code that you wrote, and start debugging from there.

Tip

If you have Products.PdbDebugMode installed, you will usually end up at a (pdb) prompt when a traceback occurs. You can use this to investigate what is going on. The up and down PDB commands can be used to move up or down the stack. Use the c command to exit the debugger.

Find the documentation

Documentation for Plone code can usually be found in one of the following places:

  • High level and tutorial-style documentation is normally found at http://plone.org/documentation.
  • Many distributions include documentation in their PyPI listings. For example, plone.testing's documentation can be found at http://pypi.python.org/pypi/plone.testing.
  • Distributions often include documentation in their README file. This is often also the source of the documentation published to PyPI.
  • Internal documentation in the form of comments or tests is often very valuable. Most packages use Zope interfaces (more on those in Chapter 9, Nine Core Concepts of Zope Programming) to describe their key APIs. Look for a file called interfaces.py.

Use the Source, Luke!

Python's readability is both a blessing and a curse. A blessing because it is normally possible to read the source code to find out what is going on. A curse because this sometimes makes developers a little lax about documentation.

One of the first hurdles new developers should overcome is any undue respect for the Python files that make up Zope and Plone. There is (almost) nothing magical about them. Earlier, we showed how to install an 'omelette' (in the parts/omelette directory inside the buildout root) that provides a single view of the source code that makes up the Plone runtime environment for a project. This is the best place to look for source code.

Get used to searching for code using grep or equivalent graphical tools, opening them and looking for specific classes and methods. Seeing what a piece of code does can often be faster than looking up documentation or examples. As time goes by, you will find that a few packages come up repeatedly, and finding code will be easier.

You can of course change these files as well. A backup is advisable, but if you think that temporarily raising an exception, adding a PDB break point, or printing a message from somewhere deep inside Zope helps you to solve a problem, go right ahead.

Note

It is a bad idea to make permanent changes this way, however, not least because those changes will be overwritten if you upgrade or reinstall the particular component. In the next chapter, we will learn more about other ways of customizing code Zope and Plone. However, if you find a bug, please report it (at http://dev.plone.org/plone), and attach a patch if you can!

Become familiar with the debugger

PDB, the Python debugger, is your best friend. To insert a breakpoint in your code or in some other code that you are trying to debug add the following line and (re-)start Zope in the foreground in a terminal:

import pdb; pdb.set_trace()

Tip

To avoid a restart, you can usually use the @@reload view from plone.reload. Refer the previous sections.

When this line is encountered, execution will stop, and the terminal will display:

(pdb)

This is the interactive PDB prompt. Type help and press Enter to see available commands. The most important ones are pp, to print a variable or the result of an expression; n to step to the next line; s to step into a function call; l to show a listing of the source code around the current execution point; tbreak to add a temporary breakpoint at a particular line, and c, to stop debugging and continue execution until another breakpoint is encountered.

If you want to quickly test syntax or libraries, you can run Python's interactive interpreter, through the bin/zopepy script as described earlier.

Look at the logs

During development, you should typically run Plone in a terminal window, using bin/instance fg. This enables debug mode and prints log messages to the console. If anything goes wrong, you should inspect the log first.

If an exception is encountered, you will likely see a traceback. For example:

Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.CMFPlone.FactoryTool, line 361, in __call__
Module Products.CMFPlone.FactoryTool, line 147, in __getitem__
Module Products.CMFPlone.PloneFolder, line 406, in invokeFactory
Module Products.CMFCore.TypesTool, line 934, in constructContent
Module Products.CMFCore.TypesTool, line 345, in constructInstance
Module Products.CMFCore.TypesTool, line 357, in _finishConstruction
Module Products.CMFCore.CMFCatalogAware, line 145, in notifyWorkflowCreated
Module Products.CMFCore.WorkflowTool, line 355, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 392, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 476, in _changeStateOf
Module Products.DCWorkflow.DCWorkflow, line 571, in _executeTransition
Module Products.DCWorkflow.DCWorkflow, line 435, in updateRoleMappingsFor
Module Products.DCWorkflow.utils, line 60, in modifyRolesForPermission
Module AccessControl.Permission, line 93, in setRoles
AttributeError: appname

The actual error is usually on the last line of the traceback. If this is not in any code that you wrote, chances are something you did further up the stack caused the problem, if you passed an invalid argument to a function. Start from the end and work your way up the stack trace until you see some code that you wrote, and start debugging from there.

Tip

If you have Products.PdbDebugMode installed, you will usually end up at a (pdb) prompt when a traceback occurs. You can use this to investigate what is going on. The up and down PDB commands can be used to move up or down the stack. Use the c command to exit the debugger.

Use the Source, Luke!

Python's readability is both a blessing and a curse. A blessing because it is normally possible to read the source code to find out what is going on. A curse because this sometimes makes developers a little lax about documentation.

One of the first hurdles new developers should overcome is any undue respect for the Python files that make up Zope and Plone. There is (almost) nothing magical about them. Earlier, we showed how to install an 'omelette' (in the parts/omelette directory inside the buildout root) that provides a single view of the source code that makes up the Plone runtime environment for a project. This is the best place to look for source code.

Get used to searching for code using grep or equivalent graphical tools, opening them and looking for specific classes and methods. Seeing what a piece of code does can often be faster than looking up documentation or examples. As time goes by, you will find that a few packages come up repeatedly, and finding code will be easier.

You can of course change these files as well. A backup is advisable, but if you think that temporarily raising an exception, adding a PDB break point, or printing a message from somewhere deep inside Zope helps you to solve a problem, go right ahead.

Note

It is a bad idea to make permanent changes this way, however, not least because those changes will be overwritten if you upgrade or reinstall the particular component. In the next chapter, we will learn more about other ways of customizing code Zope and Plone. However, if you find a bug, please report it (at http://dev.plone.org/plone), and attach a patch if you can!

Become familiar with the debugger

PDB, the Python debugger, is your best friend. To insert a breakpoint in your code or in some other code that you are trying to debug add the following line and (re-)start Zope in the foreground in a terminal:

import pdb; pdb.set_trace()

Tip

To avoid a restart, you can usually use the @@reload view from plone.reload. Refer the previous sections.

When this line is encountered, execution will stop, and the terminal will display:

(pdb)

This is the interactive PDB prompt. Type help and press Enter to see available commands. The most important ones are pp, to print a variable or the result of an expression; n to step to the next line; s to step into a function call; l to show a listing of the source code around the current execution point; tbreak to add a temporary breakpoint at a particular line, and c, to stop debugging and continue execution until another breakpoint is encountered.

If you want to quickly test syntax or libraries, you can run Python's interactive interpreter, through the bin/zopepy script as described earlier.

Look at the logs

During development, you should typically run Plone in a terminal window, using bin/instance fg. This enables debug mode and prints log messages to the console. If anything goes wrong, you should inspect the log first.

If an exception is encountered, you will likely see a traceback. For example:

Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.CMFPlone.FactoryTool, line 361, in __call__
Module Products.CMFPlone.FactoryTool, line 147, in __getitem__
Module Products.CMFPlone.PloneFolder, line 406, in invokeFactory
Module Products.CMFCore.TypesTool, line 934, in constructContent
Module Products.CMFCore.TypesTool, line 345, in constructInstance
Module Products.CMFCore.TypesTool, line 357, in _finishConstruction
Module Products.CMFCore.CMFCatalogAware, line 145, in notifyWorkflowCreated
Module Products.CMFCore.WorkflowTool, line 355, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 392, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 476, in _changeStateOf
Module Products.DCWorkflow.DCWorkflow, line 571, in _executeTransition
Module Products.DCWorkflow.DCWorkflow, line 435, in updateRoleMappingsFor
Module Products.DCWorkflow.utils, line 60, in modifyRolesForPermission
Module AccessControl.Permission, line 93, in setRoles
AttributeError: appname

The actual error is usually on the last line of the traceback. If this is not in any code that you wrote, chances are something you did further up the stack caused the problem, if you passed an invalid argument to a function. Start from the end and work your way up the stack trace until you see some code that you wrote, and start debugging from there.

Tip

If you have Products.PdbDebugMode installed, you will usually end up at a (pdb) prompt when a traceback occurs. You can use this to investigate what is going on. The up and down PDB commands can be used to move up or down the stack. Use the c command to exit the debugger.

Become familiar with the debugger

PDB, the Python debugger, is your best friend. To insert a breakpoint in your code or in some other code that you are trying to debug add the following line and (re-)start Zope in the foreground in a terminal:

import pdb; pdb.set_trace()

Tip

To avoid a restart, you can usually use the @@reload view from plone.reload. Refer the previous sections.

When this line is encountered, execution will stop, and the terminal will display:

(pdb)

This is the interactive PDB prompt. Type help and press Enter to see available commands. The most important ones are pp, to print a variable or the result of an expression; n to step to the next line; s to step into a function call; l to show a listing of the source code around the current execution point; tbreak to add a temporary breakpoint at a particular line, and c, to stop debugging and continue execution until another breakpoint is encountered.

If you want to quickly test syntax or libraries, you can run Python's interactive interpreter, through the bin/zopepy script as described earlier.

Look at the logs

During development, you should typically run Plone in a terminal window, using bin/instance fg. This enables debug mode and prints log messages to the console. If anything goes wrong, you should inspect the log first.

If an exception is encountered, you will likely see a traceback. For example:

Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.CMFPlone.FactoryTool, line 361, in __call__
Module Products.CMFPlone.FactoryTool, line 147, in __getitem__
Module Products.CMFPlone.PloneFolder, line 406, in invokeFactory
Module Products.CMFCore.TypesTool, line 934, in constructContent
Module Products.CMFCore.TypesTool, line 345, in constructInstance
Module Products.CMFCore.TypesTool, line 357, in _finishConstruction
Module Products.CMFCore.CMFCatalogAware, line 145, in notifyWorkflowCreated
Module Products.CMFCore.WorkflowTool, line 355, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 392, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 476, in _changeStateOf
Module Products.DCWorkflow.DCWorkflow, line 571, in _executeTransition
Module Products.DCWorkflow.DCWorkflow, line 435, in updateRoleMappingsFor
Module Products.DCWorkflow.utils, line 60, in modifyRolesForPermission
Module AccessControl.Permission, line 93, in setRoles
AttributeError: appname

The actual error is usually on the last line of the traceback. If this is not in any code that you wrote, chances are something you did further up the stack caused the problem, if you passed an invalid argument to a function. Start from the end and work your way up the stack trace until you see some code that you wrote, and start debugging from there.

Tip

If you have Products.PdbDebugMode installed, you will usually end up at a (pdb) prompt when a traceback occurs. You can use this to investigate what is going on. The up and down PDB commands can be used to move up or down the stack. Use the c command to exit the debugger.

Look at the logs

During development, you should typically run Plone in a terminal window, using bin/instance fg. This enables debug mode and prints log messages to the console. If anything goes wrong, you should inspect the log first.

If an exception is encountered, you will likely see a traceback. For example:

Traceback (innermost last):
Module ZPublisher.Publish, line 115, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 41, in call_object
Module Products.CMFPlone.FactoryTool, line 361, in __call__
Module Products.CMFPlone.FactoryTool, line 147, in __getitem__
Module Products.CMFPlone.PloneFolder, line 406, in invokeFactory
Module Products.CMFCore.TypesTool, line 934, in constructContent
Module Products.CMFCore.TypesTool, line 345, in constructInstance
Module Products.CMFCore.TypesTool, line 357, in _finishConstruction
Module Products.CMFCore.CMFCatalogAware, line 145, in notifyWorkflowCreated
Module Products.CMFCore.WorkflowTool, line 355, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 392, in notifyCreated
Module Products.DCWorkflow.DCWorkflow, line 476, in _changeStateOf
Module Products.DCWorkflow.DCWorkflow, line 571, in _executeTransition
Module Products.DCWorkflow.DCWorkflow, line 435, in updateRoleMappingsFor
Module Products.DCWorkflow.utils, line 60, in modifyRolesForPermission
Module AccessControl.Permission, line 93, in setRoles
AttributeError: appname

The actual error is usually on the last line of the traceback. If this is not in any code that you wrote, chances are something you did further up the stack caused the problem, if you passed an invalid argument to a function. Start from the end and work your way up the stack trace until you see some code that you wrote, and start debugging from there.

Tip

If you have Products.PdbDebugMode installed, you will usually end up at a (pdb) prompt when a traceback occurs. You can use this to investigate what is going on. The up and down PDB commands can be used to move up or down the stack. Use the c command to exit the debugger.

Summary

In this chapter, we have seen:

  • How to prepare a development environment for Zope and Plone development
  • How to set up a minimal Zope buildout and install a Plone site into it
  • How Buildout and Setuptools/Distribute work
  • How to set up a more sophisticated and flexible development buildout. This buildout will be used throughout the book
  • Some important development debugging tools
  • A few tips on how you can more effectively help yourself by learning to look at source code, using PDB effectively, prototyping things with zopepy, and looking at log files

This concludes Part 1 of the book. In Part 2, we will learn more about customizing Plone for our specific needs.

About the Authors
  • Martin Aspeli

    Martin Aspeli is an experienced Plone consultant and a prolific Plone contributor. He served on the Framework Team for Plone 3.0, and is responsible for many new features such as the improved portlets infrastructure, the “content rules” engine, and several R&D efforts relating to Plone 4.0. He is a former leader of the Plone Documentation Team and has written a number of well-received tutorials available on plone.org. He is also the author of Professional Plone Development and was recognized in 2008 by Packt Publishing as one of the “Most Valuable People” in Open source Content Management Systems.

    Browse publications by this author
  • The Plone Foundation Alex Limi Toby Roberts (Project)
Latest Reviews (1 reviews total)
This book flows logically. It is an awesome addition to any library.
Professional Plone 4 Development
Unlock this book and the full library FREE for 7 days
Start now