In this chapter, we will talk about the following topics:
The story of Django
How Django works
What is a Pattern?
Well-known pattern collections
Patterns in Django
According to Bowei Gai's "World Startup Report," there were more than 136,000 Internet firms across the world in 2013, with more than 60,000 in America alone. Of these, 87 US companies are valued more than 1 billion dollars. Another study says that of 12,000 people aged between 18 and 30 in 27 countries, more than two-thirds see opportunities in becoming an entrepreneur.
This entrepreneurial boom in digital startups is primarily due to the tools and technologies of startups becoming cheap and ubiquitous. Creating a fully fledged web application takes a lot less time than it used to, thanks to powerful frameworks.
With a gentle learning curve, even first-time programmers can learn to create web applications easily. However, soon they would keep solving the same problems others have been facing again and again. This is where understanding patterns can really help save their time.
Every web application is different, like a piece of handcrafted furniture. You will rarely find a mass-produced one meeting all your needs perfectly. Even if you start with a basic requirement, such as a blog or a social network, your needs will slowly grow, and you can easily end up with a lot of half-baked solutions duct-taped onto a once-simple cookie cutter solution.
This is why web frameworks such as Django or Rails have become extremely popular. Frameworks speed up development and have all the best practices baked in. However, they are also flexible enough to give you access to just enough plumbing for the job. Today, web frameworks are ubiquitous and most programming languages have at least one end-to-end framework similar to Django.
Python probably has more web frameworks than most programming languages. A quick look at PyPi (Python Package Index) brings up an amazing 13,021 packages related to frameworks. For Django, the total is 5,467 packages.
The Python wiki lists over 54 active web frameworks with the most popular ones being Django, Flask, Pyramid, and Zope. Python also has a wide diversity in frameworks. The compact
Bottle micro web-framework is just one Python file that has no dependencies and is surprisingly capable of creating a simple web application.
Despite these abundant options, Django has emerged as a big favorite by a wide margin. Djangosites.org lists over 4,700 sites written in Django, including famous success stories such as Instagram, Pinterest, and Disqus.
As the official description says, Django (https://djangoproject.com) is a high-level Python web framework that encourages rapid development and clean, pragmatic design. In other words, it is a complete web framework with batteries included, just like Python.
The out-of-the-box admin interface, one of Django's unique features, is extremely helpful for early data entry and testing. Django's documentation has been praised for being extremely well-written for an open source project.
Finally, Django has been battle-tested in several high traffic websites. It has an exceptionally sharp focus on security with protection against common attacks such as Cross-site scripting (XSS) and Cross-site request forgery (CSRF).
Although in theory, you can use Django to build any kind of web application, it might not be the best for every use case. For example, to build a real-time interface for web-based chat, you might want to use Tornado, while the rest of your web app can still be done in Django. Choose the right tool for the job.
Some of the built-in features, such as the admin interface, might sound odd if you are used to other web frameworks. To understand the design of Django, let's find out how it came into being.
When you look at the Pyramids of Egypt, you would think that such a simple and minimal design must have been quite obvious. In truth, they are products of 4,000 years of architectural evolution. Step Pyramids, the initial (and clunky) design, had six rectangular blocks of decreasing size. It took several iterations of architectural and engineering improvements until the modern, glazing, and long-lasting limestone structures were invented.
Looking at Django you might get a similar feeling. So, elegantly built, it must have been flawlessly conceived. On the contrary, it was the result of rewrites and rapid iterations in one of the most high-pressure environments imaginable—a newsroom!
In the fall of 2003, two programmers, Adrian Holovaty and Simon Willison, working at the Lawrence Journal-World newspaper, were working on creating several local news websites in Kansas. These sites, including LJWorld.com, Lawrence.com, and KUsports.com—like most news sites were not just content-driven portals chock-full of text, photos, and videos, but they also constantly tried to serve the needs of the local Lawrence community with applications, such as a local business directory, events calendar, classifieds, and so on.
This, of course, meant lots of work for Simon, Adrian, and later Jacob Kaplan Moss who had joined their team; with very short deadlines, sometimes with only a few hours' notice. Since it was the early days of web development in Python, they had to write web applications mostly from scratch. So, to save precious time, they gradually refactored out the common modules and tools into something called "The CMS."
Eventually, the content management parts were spun off into a separate project called the Ellington CMS, which went on to become a successful commercial CMS product. The rest of "The CMS" was a neat underlying framework that was general enough to be used to build web applications of any kind.
By July 2005, this web development framework was released as Django (pronounced Jang-Oh) under an open source Berkeley Software Distribution (BSD) license. It was named after the legendary jazz guitarist Django Reinhardt. And the rest, as they say, is history.
Due to its humble origins as an internal tool, Django had a lot of Lawrence Journal-World-specific oddities. To make Django truly general purpose, an effort dubbed "Removing the Lawrence" was already underway.
However, the most significant refactoring effort that Django developers had to undertake was called "Removing the Magic." This ambitious project involved cleaning up all the warts Django had accumulated over the years, including a lot of magic (an informal term for implicit features) and replacing them with a more natural and explicit Pythonic code. For example, the model classes used to be imported from a magic module called
django.models.*, rather than directly importing them from the
models.py module they were defined in.
At that time, Django had about a hundred thousand lines of code, and it was a significant rewrite of the API. On May 1, 2006, these changes, almost the size of a small book, were integrated into Django's development version trunk and released as Django release 0.95. This was a significant step toward the Django 1.0 milestone.
Every year, conferences called DjangoCons are held across the world for Django developers to meet and interact with each other. They have an adorable tradition of giving a semi-humorous keynote on "why Django sucks." This could be a member of the Django community, or someone who works on competing web frameworks or just any notable personality.
Over the years, it is amazing how Django developers took these criticisms positively and mitigated them in subsequent releases. Here is a short summary of the improvements corresponding to what once used to be a shortcoming in Django and the release they were resolved in:
New form-handling library (Django 0.96)
Decoupling admin from models (Django 1.0)
Multiple database support (Django 1.2)
Managing static files better (Django 1.3)
Better time zone support (Django 1.4)
Customizable user model (Django 1.5)
Better transaction handling (Django 1.6)
Built-in database migrations (Django 1.7)
To truly appreciate Django, you will need to peek under the hood and see the various moving parts inside. This can be both enlightening and overwhelming. If you are already familiar with this, you might want to skip this section.
The preceding figure shows the simplified journey of a web request from a visitor's browser to your Django application and back. The numbered paths are as follows:
The browser sends the request (essentially, a string of bytes) to your web server.
Your web server (say, Nginx) hands over the request to a WSGI server (say, uWSGI) or directly serves a file (say, a CSS file) from the filesystem.
Unlike a web server, WSGI servers can run Python applications. The request populates a Python dictionary called
environand, optionally, passes through several layers of middleware, ultimately reaching your Django application.
URLconf contained in the
urls.pyof your application selects a view to handle the request based on the requested URL. The request has turned into
HttpRequest—a Python object.
The selected view typically does one or more of the following things:
5a. Talks to a database via the models
5b. Renders HTML or any other formatted response using templates
5c. Returns a plain text response (not shown)
5d. Raises an exception
A beautifully rendered web page is seen in your user's browser.
Though certain details are omitted, this representation should help you appreciate Django's high-level architecture. It also show the roles played by the key components, such as models, views, and templates. Many of Django's components are based on several well-known design patterns.
What is common between the words "Blueprint," "Scaffolding," and "Maintenance"? These software development terms have been borrowed from the world of building construction and architecture. However, one of the most influential terms comes from a treatise on architecture and urban planning written in 1977 by the leading Austrian architect Christopher Alexander and his team consisting of Murray Silverstein, Sara Ishikawa, and several others.
The term "Pattern" came in vogue after their seminal work, A Pattern Language: Towns, Buildings, Construction (volume 2 in a five-book series) based on the astonishing insight that users know about their buildings more than any architect ever could. A pattern refers to an everyday problem and its proposed but time-tested solution.
In the book, Christopher Alexander states that "Each pattern describes a problem, which occurs over and over again in our environment, and then describes the core of the solution to that problem in such a way that you can use this solution a million times over, without ever doing it the same way twice."
For example, the Wings Of Light pattern describes how people prefer buildings with more natural lighting and suggests arranging the building so that it is composed of wings. These wings should be long and narrow, never more than 25 feet wide. Next time you enjoy a stroll through the long well-lit corridors of an old university, be grateful to this pattern.
Their book contained 253 such practical patterns, from the design of a room to the design of entire cities. Most importantly, each of these patterns gave a name to an abstract problem and together formed a pattern language.
Remember when you first came across the word déjà vu? You probably thought "Wow, I never knew that there was a word for that experience." Similarly, architects were not only able to identify patterns in their environment but could also, finally, name them in a way that their peers could understand.
In the world of software, the term design pattern refers to a general repeatable solution to a commonly occurring problem in software design. It is a formalization of best practices that a developer can use. Like in the world of architecture, the pattern language has proven to be extremely helpful to communicate a certain way of solving a design problem to other programmers.
There are several collections of design patterns but some have been considerably more influential than the others.
One of the earliest efforts to study and document design patterns was a book titled Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides, who later became known as the Gang of Four (GoF). This book is so influential that many consider the 23 design patterns in the book as fundamental to software engineering itself.
In reality, the patterns were written primarily for object-oriented programming languages, and it had code examples in C++ and Smalltalk. As we will see shortly, many of these patterns might not be even required in other programming languages with better higher-order abstractions such as Python.
The 23 patterns have been broadly classified by their type as follows:
Behavioral Patterns: These include Chain of Responsibility, Command Pattern, Interpreter Pattern, Iterator Pattern, Mediator Pattern, Memento Pattern, Observer Pattern, State Pattern, Strategy Pattern, Template Pattern, and Visitor Pattern
While a detailed explanation of each pattern would be beyond the scope of this book, it would be interesting to identify some of these patterns in Django itself:
This encapsulates a request in an object
When one object changes state, all its listeners are notified and updated automatically
Class-based generic views
Steps of an algorithm can be redefined by subclassing without changing the algorithm's structure
While these patterns are mostly of interest to those studying the internals of Django, the pattern under which Django itself can be classified under—is a common question.
Today, MVC is a very popular pattern in web application frameworks. Beginners often ask the question—is Django an MVC framework?
The answer is both yes and no. The MVC pattern advocates the decoupling of the presentation layer from the application logic. For instance, while designing an online game website API, you might present a game's high scores table as an HTML, XML, or comma-separated (CSV) file. However, its underlying model class would be designed independent of how the data would be finally presented.
MVC is very rigid about what models, views, and controllers do. However, Django takes a much more practical view to web applications. Due to the nature of the HTTP protocol, each request for a web page is independent of any other request. Django's framework is designed like a pipeline to process each request and prepare a response.
Django calls this the Model-Template-View (MTV) architecture. There is separation of concerns between the database interfacing classes (Model), request-processing classes (View), and a templating language for the final presentation (Template).
If you compare this with the classic MVC—"Model" is comparable to Django's Models, "View" is usually Django's Templates, and "Controller" is the framework itself that processes an incoming HTTP request and routes it to the correct view function.
If this has not confused you enough, Django prefers to name the callback function to handle each URL a "view" function. This is, unfortunately, not related to the MVC pattern's idea of a View.
Unlike the GoF book, which described design patterns, Fowler's book was about architectural patterns. Hence, they describe patterns at a much higher level of abstraction and are largely programming language agnostic.
Fowler's patterns are organized as follows:
Object-Relational Structural Patterns: These include Foreign Key Mapping, Mapping, Dependent Mapping, Association Table Mapping, Identity Field, Serialized LOB, Embedded Value, Inheritance Mappers, Single Table Inheritance, Concrete Table Inheritance, and Class Table Inheritance
Almost all of these patterns would be useful to know while architecting a Django application. In fact, Fowler's website at http://martinfowler.com/eaaCatalog/ has an excellent catalog of these patterns. I highly recommend that you check them out.
Django also implements a number of these patterns. The following table lists a few of them:
Encapsulates the database access, and adds domain logic on that data
Class Table Inheritance
Each entity in the hierarchy is mapped to a separate table
Saves a database ID field in an object to maintain identity
Renders into HTML by embedding markers in HTML
Yes, of course. Patterns are discovered all the time. Like living beings, some mutate and form new patterns: take, for instance, MVC variants such as Model–view–presenter (MVP), Hierarchical model–view–controller (HMVC), or Model View ViewModel (MVVM).
Patterns also evolve with time as better solutions to known problems are identified. For example, Singleton pattern was once considered to be a design pattern but now is considered to be an Anti-pattern due to the shared state it introduces, similar to using global variables. An Anti-pattern can be defined as commonly reinvented but a bad solution to a problem.
Some of the other well-known books which catalog patterns are Pattern-Oriented Software Architecture (known as POSA) by Buschmann, Meunier, Rohnert, Sommerlad, and Sta; Enterprise Integration Patterns by Hohpe and Woolf; and The Design of Sites: Patterns, Principles, and Processes for Crafting a Customer-Centered Web Experience by Duyne, Landay, and Hong.
This book will cover Django-specific design and architecture patterns, which would be useful to a Django developer. The upcoming sections will describe how each pattern will be presented.
The heading is the pattern name. If it is a well-known pattern, the commonly used name is used; otherwise, a terse, self-descriptive name has been chosen. Names are important, as they help in building the pattern vocabulary. All patterns will have the following parts:
Problem: This briefly mentions the problem.
Solution: This summarizes the proposed solution(s).
Problem Details: This elaborates the context of the problem and possibly gives an example.
Solution Details: This explains the solution(s) in general terms and provides a sample Django implementation.
Patterns compensate for the missing language features: Peter Norvig found that 16 of the 23 patterns in Design Patterns were 'invisible or simpler' in Lisp. Considering Python's introspective facilities and first-class functions, this might as well be the case for Python too.
Patterns repeat best practices: Many patterns are essentially formalizations of best practices such as separation of concerns and could seem redundant.
Patterns can lead to over-engineering: Implementing the pattern might be less efficient and excessive compared to a simpler solution.
Don't implement a pattern if your language supports a direct solution
Don't try to retro-fit everything in terms of patterns
Use a pattern only if it is the most elegant solution in your context
Don't be afraid to create new patterns
In addition to design patterns, there might be a recommended approach to solving a problem. In Django, as with Python, there might be several ways to solve a problem but one idiomatic approach among those.
Generally, the Python community uses the term 'Pythonic' to describe a piece of idiomatic code. It typically refers to the principles laid out in 'The Zen of Python'. Written like a poem, it is extremely useful to describe such a vague concept.
Furthermore, Django developers have crisply documented their design philosophies while designing the framework at https://docs.djangoproject.com/en/dev/misc/design-philosophies/.
While the document describes the thought process behind how Django was designed, it is also useful for developers using Django to build applications. Certain principles such as Don't Repeat Yourself (DRY), loose coupling, and tight cohesion can help you write more maintainable and idiomatic Django applications.
Django or Python best practices suggested by this book would be formatted in the following manner:
In this chapter, we looked at why people choose Django over other web frameworks, its interesting history, and how it works. We also examined design patterns, popular pattern collections, and best practices.
In the next chapter, we will take a look at the first few steps in the beginning of a Django project such as gathering requirements, creating mockups, and setting up the project.