In thisÂ chapter, we will talk about the following topics:
- Why Django?
- The story of Django
- How does Django work?
- What is a pattern?
- Well-known pattern collections
- Patterns in Django
AccordingÂ to Bowei Gai'sÂ World Startup Report, there were more than 136,000 internet firms across the world in 2013, with more than 60,000 in America alone. Of these, 87 US companies are valued at more than $1 billion. Another study says that of 12,000 people aged between 18 years and 30 years in 27 countries, more than two-thirds see opportunities to become an entrepreneur.
This entrepreneurial boom in digital startups is primarily due to the tools and technologies of startups becoming cheap and ubiquitous. Creating a full-fledged web application takes a lot less time and skill than it used to, thanks to powerful frameworks.
Physicists, educators, artists, and many others without a software engineering background are creating useful applications that are significantly advancing their domains. However, they may not be aware of the software engineering design principles needed to construct large and maintainable software.
A study of four different implementations of a web-based application in Norway showed implementations with known code smells and design anti-patterns to be directly associated with the difficulties in maintenance. Poorly designed software might work just as well but can be difficult to adapt to evolving requirements in a fast-changing world.
Beginners often discover design issues late in their project. Soon, they would attempt to solve the same problems others have faced again and again. This is where understanding patterns can really help save theirÂ time.
Every web application is different, like aÂ pieceÂ of handcrafted furniture. You will rarely find a mass-produced sofa meeting all your needs perfectly. Even if you start with a basic requirement, such as a blog or social network, your needs will slowly grow, and you can easily end up with a lot of half-baked solutions duct-taped onto a once simple cookie cutter solution.
This is why web frameworks, such as Django or Rails, have become extremely popular. Frameworks speed up development and have all the best practices baked in. However, they are also flexible enough to give you access to just enough plumbing for the job. Today, web frameworks are ubiquitous, and most programming languages have at least one end-to-end framework similar to Django.
Python probably has more web frameworks than most programming languages. A quick look atÂ Python Package IndexÂ (PyPI) brings up an amazing 13,045 packages related to web environments. For Django, the total is 9,091 packages. The Python wiki lists over 54 active web frameworks with the most popular ones being Django, Flask, Pyramid, and Zope. Python also has a wide diversity in frameworks. The compact bottle micro web-framework is just one Python file that has no dependencies and is surprisingly capable of creating a simple web application.
Despite these abundant options, Django has emerged as a big favorite by a wide margin.Â Djangosites.orgÂ lists over 5,263 sites written in Django, includingÂ famous success stories such as Instagram, Pinterest, and Disqus. As theÂ officialÂ description says, Django (https://djangoproject.com) is a high-level Python web framework that encourages rapid development and clean, pragmatic design. In other words, it is a complete web framework with batteries included just like Python.
The out-of-the-box admin interface, one of Django's unique features, is extremely helpful for early data entry and administration. Django's documentation has been praisedÂ for being extremely well-written for an open source project.
Finally, Django has been battle-tested inÂ severalÂ high traffic websites. It has an exceptionally sharp focus on security with protection against common attacksÂ such asÂ Cross-site scriptingÂ (XSS),Â Cross-site request forgeryÂ (CSRF) to evolving security threats such as weak password hashing algorithms.
Although you can use Django to build anyÂ kindÂ of web applicationÂ in theory, it might not be the best for every use case.Â For example, to prototype a simple web service in an embedded system with tight memory constraints, you might want to use Flask, while you might eventually move toÂ Django for its robustness and features. Choose the right tool for the job.
Some of the built-in features, such as the admin interface, might sound odd if youÂ are used to other web frameworks. To understand the design of Django, let's find out how it came into being.
When you look at the Pyramids of Egypt, youÂ wouldÂ think that such a simple and minimal design must have been quite obvious. In truth, they are the products of 4,000 years of architectural evolution. Step Pyramids, the initial (and clunky) design, had six rectangular blocks of decreasing size. It took several iterations of architectural and engineering improvements until the modern, glazing, and long-lasting limestone structures were invented.
Looking at Django, you might get a similar feeling â so elegantly built, it must have been flawlessly conceived. On the contrary, it was the result of rewrites and rapid iterations in one of the most high-pressure environments imaginableÂ â a newsroom!
In the fall of 2003, two programmers, Adrian Holovaty and Simon Willison, working at theÂ Lawrence Journal-WorldÂ newspaper, were working on creating several local news websites in Kansas. These sites, includingÂ LJWorld.com,Â Lawrence.com, andÂ KUsports.com, like most news sites were not just content-driven portals chock-full of text, photos, and videos, but they also constantly tried to serve the needs of the local Lawrence community with applications, such as a local business directory, events calendar, and classifieds.
This, of course, meant lots of work for Simon, Adrian, and later Jacob Kaplan Moss who had joined their team; with very short deadlines, sometimes with only a few hours' notice. Since it was the early days of web development in Python, they had toÂ writeÂ web applications mostly from scratch. So, to save precious time, they gradually refactored out the common modules and tools into something calledÂ TheÂ CMS.
Eventually, the content management parts wereÂ spunÂ off into a separate project called theÂ Ellington CMS, which went on to become a successful commercial CMS product. The rest of The CMS was a neat underlying framework that was general enough to be used to build web applications of any kind.
By July 2005, this web development frameworkÂ wasÂ released as Django (pronounced Jang-Oh) under an open sourceÂ Berkeley Software DistributionÂ (BSD) license.Â It was named after theÂ legendary jazz guitarist Django Reinhardt. And the rest,Â as they say, is history.
Due to its humble origins as an internal tool, Django had a lot of Lawrence Journal-World-specific oddities. To make Django truly general purpose, an effort dubbedÂ Removing the LawrenceÂ had alreadyÂ beenÂ underway.
However, the most significant refactoring effort thatÂ DjangoÂ developers had to undertake was calledÂ Removing the Magic.Â This ambitious project involved cleaning up all the warts Django had accumulated over the years, including aÂ lot of magic (an informal term for implicit features) and replacing them with aÂ more natural and explicit Pythonic code. For example, the model classes usedÂ to be imported from a magic module calledÂ
django.models.*, rather thanÂ beingÂ directly imported from theÂ
models.pyÂ module they were defined in.
At that time, Django had about a hundred thousand lines of code, and it was a significant rewrite of the API. On May 1, 2006, these changes, almost the size of a small book, were integrated into Django's development version trunk and released as Django release 0.95. This was a significant step toward the Django 1.0 milestone.
Every year, conferences calledÂ DjangoConsÂ are heldÂ acrossÂ the world for Django developers to meet and interact with each other. They have an adorable tradition of giving a semi-humorous keynote onÂ why Django sucks. This could be a member of the Django community, or someone who works on competing web frameworks orÂ justÂ any notable personality.Â Over the years, it is amazing how Django developers took these criticismsÂ positively and mitigated them in subsequent releases.
Here is a short summaryÂ of the improvements corresponding to what once used to be a shortcoming inÂ Django and the release they were resolved in:
- New form-handling library (Django 0.96)
- Decoupling admin from models (Django 1.0)
- Multiple database supports (Django 1.2)
- Managing static files better (Django 1.3)
- Better time zone support (Django 1.4)
- Customizable user model (Django 1.5)
- Better transaction handling (Django 1.6)
- Built-in database migrations (Django 1.7)
- Multiple template engines (Django 1.8)
- Simplified URL routing syntax (Django 2.0)
Over time, Django has become one of most idiomatic Python codebases in the public domain. Django source code is also a great place to learn the architecture of a large Python web framework.
To truly appreciate Django, you will need toÂ peekÂ under the hood and see the various moving parts inside. This can be both enlightening and overwhelming. If you are already familiar with the following information, you might want to skip this section:
How web requests are processed in a typical Django application
The preceding diagram shows the simplified journey of a web request from a visitor's browser to your Django application and back. The numbered paths are as follows:
- The browser sends the request (essentially, a string of bytes) to your web server.
- Your web server (say, Nginx) handsÂ overÂ the request to aÂ Web Server Gateway InterfaceÂ (WSGI) server (say, uWSGI) or directly serves a file (say, a CSS file) from the filesystem.
- Unlike a web server, WSGI servers can run Python applications. The request populates a Python dictionary calledÂ
environÂ and, optionally, passes through several layers of middleware, ultimately reaching your Django application.
- URLconf (URL configuration) module contained inÂ theÂ
urls.pyÂ of your project selects a view to handle the request based on the requested URL. The request has turned intoÂ
HttpRequest, a Python object.
- The selected view typically does one or more of the following things:
a. Talks to a database via the models
b. Renders HTML or any other formatted response using templates
c. Returns a plain text response (not shown)
d. Raises an exception
HttpResponseÂ object gets rendered into a string, as it leaves the Django application.
- A beautifully rendered web page is seen in your user's browser.
Though certain details are omitted, this representation should help you appreciate Django's high-level architecture. It also shows the roles played by the key components, such as models, views, and templates. Many of Django's components are based on several well-known design patterns.
What is common between the wordsÂ blueprint,Â scaffolding, andÂ maintenance? These software development terms have been borrowed from the world of building construction and architecture. However, one of the most influentialÂ termsÂ comes from a treatise on architecture and urban planning written in 1977 by the leading Austrian architect Christopher Alexander and his team consisting of Murray Silverstein, Sara Ishikawa, and several others.
The term pattern came in vogue after their seminal work,Â A Pattern Language: Towns, Buildings, ConstructionÂ (volume 2 in a five-book series), based on the astonishing insight that users know about their buildings more than any architect ever could. A pattern refers to an everyday problem andÂ itsÂ proposed butÂ time-tested solution.
In the book, Christopher Alexander states the following:
"Each pattern describes a problem, which occurs over and over again in our environment, and then describes theÂ core of the solution to that problem in such a way that you can use this solution a million times over, without ever doing it the same way twice."
For example, hisÂ wings of lightÂ pattern describe how people prefer buildings with more natural lighting and suggests arranging the building so that it is composed of wings. These wings should be long and narrow, never more than 25 feet wide. Next time you enjoy a stroll through the long well-lit corridors of an old university, be grateful to this pattern.
Their book contained 253 such practical patterns, from the design of a room to the design of an entire city. Most importantly, each of these patterns gave a name to an abstract problem and together formed aÂ pattern language.
Remember when you first came across the wordÂ dÃ©jÃ vu? You probably thought:Â "wow, I never knew that there was a word for that experience."Â Similarly, architects were not only able to identify patterns in their environment but could also, finally, name them in a way that their peers could understand.
In the world of software, the termÂ design patternÂ refers to a general repeatable solution to a commonly occurring problem in software design. It is a formalizationÂ of best practices that a developer can use. Like in the world of architecture, the pattern language has proven to be extremely helpful to communicate a certainÂ wayÂ of solving a design problem to other programmers.
There are several collections of design patterns, but some have been considerably more influential than the others.
One of the earliest efforts to study and documentÂ designÂ patterns was a book titledÂ Design Patterns: Elements of Reusable Object-Oriented SoftwareÂ byÂ Erich Gamma,Â Richard Helm,Â Ralph Johnson, andÂ John Vlissides, who later became known as theÂ Gang of FourÂ (GoF). This book is so influential that many consider the 23Â designÂ patterns in the book as fundamental to software engineering itself.
In reality, the patterns were written primarily for static object-oriented programming languages, and it had code examples in C++ and Smalltalk. As we will see shortly, some of these patterns might not evenÂ beÂ required in other programming languages with better higher-order abstractions such as Python.
The 23 patterns have been broadly classified by their type as follows:
- Structural patterns: TheseÂ includeÂ adapter pattern, bridge pattern, composite pattern, decorator pattern, facade pattern, flyweight pattern, and proxy pattern
- Behavioral patterns: TheseÂ includeÂ chain-of-responsibility, command pattern, interpreter pattern, iterator pattern, mediator pattern, memento pattern, observer pattern, state pattern, strategy pattern, template pattern, andÂ visitor pattern
While a detailed explanation of each pattern would be beyond the scope of this book, it would be interesting to identify some of these patterns present in Django implementation itself:
This encapsulates a request in an object
When one object changes state, all its listeners are notified and updated automatically
Class-based generic views
Steps of an algorithm can be redefined by subclassing without changing the algorithm's structure
While these patterns are mostly of interest to those studying the internals ofÂ Django, the most commonly question asked is,Â under which patternÂ is Django itself classified?
Model-View-ControllerÂ (MVC) is an architecturalÂ patternÂ invented by Xerox PARC in the 70s. Being the framework used toÂ buildÂ user interfaces in Smalltalk, it gets an early mention in the GoF book.
Today, MVC is a very popular pattern in web application frameworks. A variant of the common question is whether Django is an MVC framework.
The answer is both yes and no. The MVC pattern advocates the decoupling ofÂ the presentation layer from the application logic. For instance, while designingÂ an online game website API, you might present a game's high scores table as an HTML, XML, orÂ comma-separatedÂ valuesÂ (CSV) file. However, its underlying model class would be designedÂ independentlyÂ of how the data would be finally presented.
MVC is very rigid about what models, views, and controllers do. However, Django takes a much more practical view to web applications. Due to the nature of theÂ HTTP protocol, each request for a web page is independent of any other request. Django's framework is designed like a pipeline to process each request and prepareÂ a response.
Django calls this theÂ Model-Template-ViewÂ (MTV) architecture. There isÂ a separationÂ of concerns between the database interfacing classes (model), request-processing classes (view), and a templating language for theÂ finalÂ presentation (template).
If you compare this with the classic MVC â aÂ model isÂ comparable to Django's Models; a view is usually Django's Templates, and the controller is the framework itself that processes an incoming HTTP request and routes it to the correct view function.
If this has not confused you enough, Django prefers to name the callback function to handle each URL a view function. This is, unfortunately, not related to the MVC pattern's idea of a view.
Unlike the GoF book, which described design patterns, Fowler's book was about architectural patterns. Hence, they describeÂ patternsÂ at a much higher level of abstraction and are largely programming language agnostic.
Fowler's patterns are organized as follows:
- Domain logic patterns: These includeÂ domainÂ model, transaction script, service layer, and table module
- Data source architectural patterns: These includeÂ rowÂ data gateway, table data gateway, data mapper, and active record
- Object-relational behavioral patterns: TheseÂ includeÂ Identity Map, Unit of Work, and Lazy Load
- Web presentation patterns: TheseÂ includeÂ Page Controller, Front Controller, Model View Controller, Transform View, Template View, Application Controller, and Two-Step View
- Distribution patterns: TheseÂ includeÂ Data Transfer Object and Remote Facade
- Offline concurrency patterns: TheseÂ includeÂ Coarse-Grained Lock, Implicit Lock, Optimistic Offline Lock, and Pessimistic Offline Lock
- Session state patterns: TheseÂ includeÂ Database Session State, Client Session State, and Server Session State
- Base patterns: TheseÂ includeÂ Mapper, Gateway, Layer Supertype, Registry, Value Object, Separated Interface, Money, Plugin, Special Case, Service Stub, and Record Set
Almost all of these patterns would be useful to know while architecting a Django application. In fact, Fowler's website atÂ http://martinfowler.com/eaaCatalog/Â has an excellent catalog of these patterns online. I highly recommend that you check them out.
Django also implements a number of these patterns. The following table lists a few of them:
Encapsulate the database access and add domain logic on that data
Class table inheritance
Each entity in the hierarchy is mapped to a separate table
Saves a database ID field in an object to maintain identity
RenderÂ into HTML by embedding markers in HTML
Yes, of course. PatternsÂ areÂ discovered all the time. Like living beings, some mutate and form new patterns, for instance, MVC variants such asÂ Model-view-presenterÂ (MVP),Â Hierarchical model-view-controllerÂ (HMVC), orÂ Model View ViewModelÂ (MVVM).
Patterns also evolve with time, asÂ betterÂ solutions toÂ knownÂ problems are identified. For example, Singleton pattern was once considered to be a design pattern butÂ nowÂ is considered to be anÂ anti-patternÂ due to the sharedÂ stateÂ it introduces, similar to using global variables. AnÂ anti-patternÂ can be defined as a commonly reinventedÂ but a badÂ solutionÂ to a problem.Â Some of the other well-known books that catalog patterns areÂ Pattern-oriented software architectureÂ (POSA) by Buschmann, Meunier, Rohnert, Sommerlad, and Sta;Â Enterprise Integration PatternsÂ byÂ HohpeÂ and Woolf; andÂ The Design of Sites: Patterns, Principles, and Processes for Crafting a Customer-Centered Web ExperienceÂ byÂ Duyne,Â Landay,Â and Hong.
The heading is theÂ patternÂ name. If it is a well-known pattern, the commonly used name is used; otherwise, a terse, self-descriptive name has been chosen. Names are important, as they help in building the pattern vocabulary. All patterns will haveÂ the following parts:
- Problem: This briefly mentions the problem
- Solution: This summarizes the proposed solution(s)
- Problem Details: This elaborates the context of the problem and possibly gives an example
- Solution Details: This explains the solution(s) in general terms and provides a sample Django implementation
- Patterns compensate for the missing language features: Peter Norvig found that 16 of the 23 patterns in design patterns were invisible or simpler in dynamic languages such as Lisp or Python. For instance, as functions are already objects in Python, it would be unnecessary to create separate classes to implement strategy patterns.
- Patterns repeat best practices: Many patterns are essentially formalizations of best practices, such as separation of concerns, and could seem redundant.
- Patterns can lead to over-engineering: Implementing the pattern might be less efficient and excessive compared to a simpler solution.
- Patterns are best used to communicate that you are following a well-understood design approach
- Don't implement a pattern if your language supports a direct solution
- Don't try to retrofit everything in terms of patterns
- Use a pattern only if it is the most elegant solution in your context
- Don't be afraid to create new patterns
Generally, the Python communityÂ usesÂ the termÂ PythonicÂ to describe aÂ pieceÂ of idiomatic code. It typically refers to the principles laid out inÂ The Zen of Python. Written like a poem, it is extremely useful to describe such a vague concept.
Furthermore, Django developers have crisply documented their design philosophies while designing the framework atÂ https://docs.djangoproject.com/en/dev/misc/design-philosophies/.
While the document describes the thoughtÂ processÂ behind how Django was designed, it is also useful for developers using Django to build applications. Certain principles such asÂ Don't Repeat YourselfÂ (DRY),Â loose coupling, andÂ tight cohesionÂ can help you writeÂ moreÂ maintainable and idiomatic Django applications.
In this chapter, we looked at why people choose Django over other web frameworks, its interesting history, and how it works. We also examined design patterns, popular pattern collections, and best practices.
In the next chapter, we will take a look at the first few steps in the beginning of a Django project, such as gathering requirements, creating mockups, and setting up the project.