So, what does this bigger picture look like? There are three easily identifiable areas of focus, with a fourth (call it user interaction) that either weaves through the other three or is broken down into its own groups.

Software engineering must pay heed to standards, especially non-technical (business) ones, as well as to best practices. These may or may not be followed but, since they are standards or best practices for a reason, not following them is something that should always be a conscious (and defensible) decision. It’s not unusual for business process standards and practices to span multiple software components, which can make them difficult to track if a certain degree of discipline and planning isn’t factored into the development process to make them more visible. On the purely development-related side, standards and best practices can drastically impact the creation and upkeep of code, its ongoing usefulness, and even just the ability to find a given chunk of code, when necessary.

It’s rare for code to be written simply for the sake of writing code. There’s almost always some other value associated with it, especially if there’s business value or actual revenue associated with a product that the code is a part of. In those cases, understandably, the people who are paying for the developmental effort will be very interested in ensuring that everything works as expected (code quality) and can be deployed when expected (process predictability).

The remaining policy and procedure-related concerns are generally managed by setting up and following various standards, processes, and best practices during the startup of a project (or perhaps a development team). Those items — things such as setting up source control, having standard coding conventions, and planning for repeatable, automated testing — will be examined in some detail in later chapters. Ideally, once these kinds of development processes are in place, the ongoing activities that keep them running and reliable will just become habits, a part of the day-to-day process, almost fading into the background.

Finally, with more of a focus on the code side, software engineering must, by necessity, pay heed to entire systems, keeping a universal view of the system in mind. Software is composed of a lot of elements that might be classified as atomic: they are indivisible units in and of themselves, under normal circumstances. Just like their real-world counterparts, when they start to interact, things get interesting, and hopefully useful. Unfortunately, that’s also when unexpected (or even dangerous) behaviors — bugs — usually start to appear.

This awareness is, perhaps, one of the more difficult items to cultivate. It relies on knowledge that may not be obvious, documented, or readily available. In large or complex systems, it may not even be obvious where to start looking, or what kinds of questions to ask to try to find the information needed to acquire that knowledge.

There can be as many distinct questions that can be asked about any given chunk of code as there are chunks of code to ask about — even very simple code, living in a complex system, can raise questions in response to questions, and more questions in response to those questions.

If there isn’t an obvious starting point, starting with the following really basic questions is a good first step:

1. Who will be using the functionality?
2. What will they be doing with it?
3. When, and where, will they have access to it?
4. What problem is it trying to solve? For example, why do they need it?
5. How does it have to work? If detail is lacking, breaking this one down into two separate questions is useful:
   • What should happen if it executes successfully?
   • What should happen if the execution fails?

Teasing out more information about the whole system usually starts with something as basic as the following questions:

• What other parts of the system does this code interact with?
• How does it interact with them?

Having identified all of the moving parts, thinking about “What happens if…” scenarios is a good way to identify potential points where things will break, risks, and dangerous interactions. You can ask questions like these to start discovering these points in the code:

• What happens if this argument, which expects a number, is handed a string?
• What happens if that property isn’t the object that’s expected?
• What happens if some other object tries to change this object while it’s already being changed?

Whenever one question has been answered, simply ask, What else? This can be useful for verifying whether the current answer is reasonably complete.

When the work on this book started, Python 3.11 was selected as the language version that code would be written in. There were several reasons for that selection, but the primary one was that it had been available for long enough to be in a security maintenance cycle, and with no expectation of significant changes other than for security issues. That was early in 2024, and support for the version is, as the book is being finished, going to continue for another two years, ending in October of 2027. Hand in hand with that was the fact that the bulk of the work I was doing was writing and maintaining AWS Lambda Functions written in Python, and although the 3.12 version was available for those purposes, it was new enough that I didn’t have a lot of hands-on experience with it in that context.

The Python Software Foundation, the maintainers of the language, have historically been very good about publishing comprehensive documentation of the changes to the language as each new minor version is released. Their documentation for any given version can be found online at https://www.python.org/doc/versions/, and is updated as each new version is released, including the patch versions.

Python’s release schedule and support processes are well documented online (see https://devguide.python.org/versions/), with a cadence of approximately five years between the initial release of a minor version and its official end of life. In more recent versions, the first two years or so of a given version’s life include a bugfix support phase, and the remaining three years are limited to security support. Online providers will frequently follow the life cycle of any given Python version with respect to their support of that version, though they may allow continued use of a version that has officially reached its end of life. By way of example, Amazon Web Services’ (AWS’) Python version support, as of the end of 2025, included Python 3.9, with an official end of life in October of 2025, was a supported version, with its deprecation planned in December 2025, and limitations on the ability to create or update Lambda Functions using that version starting early in 2026.

So, taking all of these factors into consideration, the determination of what Python version to use for any given project is a decision that needs to be made based on a number of factors:

• As a general rule, the most recent version available may be preferable, simply because it will provide the most current capabilities that the language provides, and will have the longest lifespan.
• Restrictions or constraints by a provider may limit those options: Using AWS as an example again, they appear to start supporting a given version of Python about a month after it is released, but that may not always be the case. Other providers may have different limitations, or it may be that the OS where development work is being done imposes its own limitation (not uncommon for Linux systems that use Python for key applications or tools).
• Even if a just-released new version is supported, there is some risk assessment that should be undertaken: The bugfix phase of the version will last for two years, and there is some risk that bug fixes implemented during that period will be problematic. The same holds true for versions that are in the security phase of their lives, though generally those have fewer issues on a release-by-release basis, likely because most of the bugs have already been dealt with by then.
• Checking the Changelog for a given version (https://docs.python.org/release/3.11.13/whatsnew/changelog.html, for example, for 3.11), or the release notes (https://docs.python.org/3/whatsnew/3.11.html for 3.11 again) may also be useful if there are specific features that are needed or desired in the project, that may not be available or implemented as expected.

As of the end of 2025, assuming that the preference is to avoid versions in their bugfix phase, the most current version that fits that criteria is 3.12. Version 3.13 would be viable with the same considerations driving version selection by the end of 2026, and 3.14, expected to be released in October 2025, would be viable by the end of 2027.

Installation of Python varies a bit across operating systems. For macOS and Windows systems, there are installers available for download on the Python website (https://www.python.org/downloads/) that generally take care of everything needed. Those installers include every version of the language going back as far as 2.0.1.

For Linux systems, installation can be more problematic. Many Linux systems have Python installed by default, because some of their programs make use of it. In several cases, those distributions also limit the available versions of Python that can be installed, in order to prevent a user from accidentally breaking system components or programs. Checking the default software installation tools in a Linux distribution is always the best first step: If the specific version of Python that is needed/desired is available, installing it using the standard tools for the distribution is the safest process. The managers of the package repository for the distribution will have made reasonable efforts to prevent installations that can break a system from being available, and anything that is available as an option would be expected to be safe.

In cases where there are no alternative versions available, there are still options worth considering before going down the path of installing from the Python website, or building from source: pyenv and Homebrew.

pyenv (https://github.com/pyenv/pyenv) is a command-line tool that allows a user to install, manage, and switch between different versions of Python at will. While switching between versions has many of the same risks as installing other versions, pyenv's ability to download and install minimal, usable Python versions without interfering with the system-level installations makes it a very good candidate for cases where project- and system-level Python version requirements conflict with each other. pyenv is also recognized by at least one Python project management tool, pipenv, and integrated well enough that the creation of a new project environment with a new Python version is handled by the project manager almost seamlessly.

Homebrew (https://brew.sh/) is a more general-purpose software management tool, capable of installing various Python versions as well as many other software packages. It is not available for Windows, at least as of late 2025, but is a viable option for macOS and Linux systems where multiple Python versions are needed.

If neither of those options is workable, falling back to downloading from the Python website is always possible too. Because of the widely varied package structures across the Linux ecosystem, those downloads are the source code and would require building the local installation from scratch. If this path must be taken, it is important to find a good, step-by-step breakdown of the processes and prerequisites involved. Usually a search along the lines of {your Linux distro name} build python 3.13 from source will yield several options.

The code in the first edition of this book was written in Python 3.6, with an eye toward compatibility with 3.7. The code in this edition was written in Python 3.11, and by the time it sees print, Python 3.14 will have been released. Over each of those versions, there have been a number of changes, many of which will not come into play in the balance of this book, but are, nonetheless, worth knowing about.

Python 3.6

Python 3.6 introduced f-strings: strings that allow variable names and operations to be defined within them to be interpreted at run time. F-strings are used frequently in the code in this edition, particularly in log messages where error type names and values are expected to change. For example, here is an error-handling block that catches errors of any Exception type (the error), with any message, and logs the error type and message, along with the current variables (vars()) when the error is caught:

except Exception as error:
    logger.exception(
        f'{error.__class__.__name__}: {error} '
        'occurred in function_name'
    )
    logger.error('inputs: {vars()}')

Other changes included:

• Allowing underscores to be used in numeric values as readability aids (for example, 1_000_000 instead of 1000000)
• Allowing type annotations to be applied to variables (for example, my_var: int = 12 instead of my_var = 12)
• Support for asynchronous use of generators (https://wiki.python.org/moin/Generators) and comprehensions (https://docs.python.org/3.11/tutorial/datastructures.html#list-comprehensions)

Python 3.7

Perhaps the most significant addition to the language in 3.7 was the addition of the dataclasses module (https://docs.python.org/3.7/library/dataclasses.html), providing a standard mechanism for defining classes whose primary intention is the storage of data, and tools for working with them. A simple dataclass, representing a person, might look like this:

from dataclasses import dataclass, asdict
@dataclass
class Person:
    given_name: str
    family_name: str
>>> ridcully = Person('Mustrum', 'Ridcully')
>>> print(ridcully)
Person(given_name='Mustrum', family_name='Ridcully')
>>> print(asdict(ridcully))
{'given_name': 'Mustrum', 'family_name': 'Ridcully'}

Other new features and improvements for this release are listed at https://docs.python.org/3/whatsnew/3.7.html.

Python 3.8

The walrus operator (:=), more formally known as assignment expressions, became available, allowing the assignment of values to variables as part of a larger, frequently conditional expression.

For example, taken from the What’s New In Python 3.8 page:

if (nlen(a) := ) > 10:
    print(
        f"List is too long ({n} elements, expected <= 10)"
    )

…allows the call to len(a) to be used once, assigning the result to n as part of the expression, rather than requiring it to be called again in the print statement.

Other new features and improvements are listed for this release at https://docs.python.org/3/whatsnew/3.8.html.

Python 3.9

New operators for dictionary types were introduced that allow merging (|) and updating (|=) dictionaries inline.

Other new features and improvements are listed for this release at https://docs.python.org/3/whatsnew/3.9.html.

Python 3.10

This release introduced structural pattern matching as an alternative to a series of if…elif… structures. For example:

match subject:
    case <pattern_1>:        #  if subject == <pattern_1>
        <action_1>           #    <action_1>
    case <pattern_2>:        #  if subject == <pattern_2>
        <action_2>           #    <action_2>
    case <pattern_3>:        #  if subject == <pattern_3>
        <action_3>           #    <action_3>
    case _:                  #  else:
        <action_wildcard>    #      <fallback

Other new features and improvements are listed for this release at https://docs.python.org/3/whatsnew/3.10.html.

Python 3.11

The major change in this release was an improvement to execution speed, ranging from 10 to 60 percent faster. A less obvious, but still significant, change was the deprecation of 22 modules, mostly concerned with older data formats no longer in common use. Of those, nine had one or more replacements already defined.

Other new features and improvements are listed for this release at https://docs.python.org/3/whatsnew/3.11.html.

Changes in Python 3.12 onwards

Much of the focus in the more recent releases has been on reducing the requirement for the Global Interpreter Lock (GIL) in Python’s interpreter. The GIL is a mechanism that allows only one native processor thread to execute Python bytecode at any given time. While that lends a considerable amount of stability to running Python code, by preventing several types of memory-related issues, it also prevents truly parallel execution of Python code. In 3.12, the GIL was attached to individual interpreters, allowing sub-processes that run in their own sub-interpreters to have their own GIL, allowing for more true parallelism. The 3.13 release provided an experimental free-threaded build mode, allowing the GIL to be disabled entirely for even more truly parallel processing capabilities.

Another change to keep an eye on is the experimental Just-in-Time (JIT) compiler, which appears to be (potentially) laying the groundwork for translating Python’s bytecode to machine code.

These changes are the ones that I, personally, have found the most significant (or at least the most interesting) and do not represent anywhere near all of the changes that were made for each of these releases. These changes are, obviously, strictly limited to those that have happened to the language itself. There are other changes that have occurred in the time between the first edition and this edition that have little to do with the language, but with how the code gets written. Of those, the one that has made the most waves is undoubtedly the advent of AI for code generation.

As Chapters 13 and 14 were in progress, there were significant advances in various Large Language Model (LLM) tools toward writing and maintaining code. An LLM is a type of Artificial Intelligence (AI), trained on massive datasets to learn patterns and rules of language. Some of the more well-known LLMs as of 2025 include OpenAI’s GPT, Anthropic’s Claude, and xAI’s Grok. While the initial focus of LLMs was more toward natural human languages like English, it was just a matter of time before the idea of similar training for programming languages was considered and added into the mix. Early code generation using LLMs showed some promise, but was typically limited to generating small, independent chunks of code, with little to no reliability for integrating those small code blocks into larger-scoped projects. Over and above that, the code generated was not always reliably representative of what the intent was, as expressed to the LLM through a supplied prompt.

By the beginning of the second quarter of 2025, though, things had improved considerably. By way of example, when CodeGPT (https://chatgpt.com/g/g-cksUvVWar-code-gpt-python-java-c-html-javascript-more) was given this prompt:

I am writing a Python package as an example of AI-generated code. I need a project starting-point that I can download that follows a standard src-directory project structure, as well as having a tests directory with subdirectories for unit tests (unit), integration tests (integration) and system tests (system). Under the src directory, there need to be directories representing a package namespace “hms.core”, and there should be corresponding unit-test directories, each prefixed with “test_” for each level of the main namespace under the src directory. The project will use Pipenv to manage package dependencies, and should include, as development package requirements, the pytest, flake8, and coverage packages. Those packages’ versions should be pinned to versions less than the next major version; for example, the current version of pytest is 8.3.5, so the pytest installation in the Pipfile should indicate a version less than 9. The package project will eventually be published as a standard Python package, so there should be a pyproject.toml file that captures all the standard information and needs to accomplish that, and package installations for the “build” and “publish” categories of the Pipfile should include the “build” and “twine” packages, respectively. The project will eventually use some cloud- or SCM-resident CI/CD process, but for now, capture the necessary steps and processes in a standard Makefile that will eventually be used to create the final build-and-publish process. That Makefile should include running unit tests and all the other test-suites noted earlier.

Please generate this project structure, and provide it as a downloadable file, like a ZIP archive or a TAR file.

…the resulting project structure was reasonably complete, and was created in less than a minute, with perhaps five to ten minutes needed to write the entire prompt. Iterating over that initial project structure was possible, starting with the addition of functionality using this prompt:

Add a data_object.py module to the hms/core directory. This module will contain a class called BaseDataObject, which will use Pydantic Fields to define various properties. Those properties include “oid” an object ID, which will be a UUID value, providing the unique record identifier for objects in a relational database table later on. It will also include “created” and “modified”, both UTC dates, capturing the created date/time and last modified date/time of the object’s record. It will also include boolean fields called “is_active” and “is_deleted” which keep track of whether the record for a data object is active and deleted, respectively. These fields are all required. The oid field should default to a new UUID value, which can use the built-in uuid4 function. The date/time fields should default to the current UTC date/time. The is_active and is_deleted fields should default to False. All fields should have a description and two or more examples. The module should have an overall description.

BaseDataObject will also provide methods to perform CRUD operations: A get method will be used to retrieve one or more objects, and accept zero-to-many oid values (strings or UUIDs), include pagination control parameters that will be used to create a SQL “LIMIT” clause, and “criteria” that will be used to provide other selection criteria, based on equality, inequality, less-than/greater-than, and so on.

Update the project code and provide a new download with those changes.

…generated a reasonable starting point for the BaseDataObject and its get method, and continuing with:

The get method should be a class method, and should assume the use of parameterized queries with an execute method, following the Python database access API standards. Add a delete class method that accepts one-to-many oid values, that simulates the deletion of one to many records, and an instance method to save an instance’s data as well.

…also yielded reasonable starting-point code for those additions.

Different AIs will generate different results, and have different ways of presenting those results. For example, the same initial prompt, given to Anthropic’s Claude (https://claude.ai), yielded a Python script that generated the project structure, and the subsequent prompt that GPT Code used to add the get method Claude used to also generate the create, update, and delete methods.

This sort of iterative, AI-assisted code generation has come to be known as vibe coding (https://en.wikipedia.org/wiki/Vibe_coding), and was a highly contentious topic in April of 2025, judging by the amount and types of discussions around it on LinkedIn and elsewhere on the Internet. The arguments for it include increased speed of output, which is certainly the case, and that the generated code is good enough, which is a very subjective judgment. The arguments against it usually centered around inconsistent and unpredictable code quality, the sheer volume of code that would have to be reviewed by a human engineer, and the lack of decision-making context that a human engineer just knows to keep in mind — things like paying heed to security concerns, which CodeGPT had to be told about (…should assume the use of parameterized queries with an execute method, following the Python database access API standards… in the second prompt above). Another concern, though its impact will vary from LLM to LLM, is the eventual loss of context by the LLM, leading to hallucinations: the generation of incorrect, misleading, or even nonsensical responses and information, typically with no warning or indication that it is happening. This is, ultimately, a function of how much context data a given LLM can keep track of, and that is bound to improve over time, but is a limiting factor whose scope may not be known.

The availability of AI agents — programs that leverage a backing AI system, but are able to keep track of the relevant context they are working in, make decisions, and take actions on their own to achieve specific goals — is another factor worth knowing about. As of this writing, I can only speak to one from personal experience: Claude Code. Claude Code appears able to keep track of a lot of context while it is working; I have used it to generate reasonably complete and functional code for creating and managing a REST API using AWS API Gateway that calls SageMaker endpoints, along with the Infrastructure as Code (IaC) to manage that API and the SageMaker endpoints it calls. Getting to that point required several days’ worth of prompt-writing and iteration over bugs that surfaced, and those efforts cost maybe $35 in Claude token purchases, but the experience was reasonable, most of the time, and the code generated was functional, if brute-forced and occasionally ugly.

Points to consider when using AI code assistants

Stick to one. Don’t change to another too frequently, even if there’s something much newer and better. Each LLM tends to have its own code style, and shifting between those is likely to be problematic over time.

Iterate in small chunks, and commit changes frequently. There are numerous stories being recounted online along the lines of [insert AI name here] destroyed my project, but the common thread among those that I’ve seen is a lack of tool and code discipline as changes are being made.

Decide on a trust boundary. Depending on how pessimistic you are, either don’t trust the AI-generated code at all, or trust it but verify it. In either case, be prepared to review a lot of auto-generated code, and tell the AI how to fix things that you do not like or agree with.

Be as specific as you can. Tell the AI what you want to accomplish, how it needs to be accomplished, what constraints are in play, etc., etc.

Set limitations on the AI. If there is code that should not be changed, tell the AI that it is not allowed to change that code, and stick to that. Similarly, apply standards as you see fit, even if they are just documentation standards.

Is AI here to stay?

If I’m being brutally honest about this, my answer would have to be some variant of I just don’t know. The ability to generate large blocks of production-ready code quickly, effectively, and consistently would have to be among the highest-priority goals in that space. From what I’ve seen personally, the systems just aren’t there yet, not without a lot of review and input from an experienced, competent software engineer. Of those three points, the consistency feels to me like it’s the most challenging problem still to be solved. I’m not surprised by that, though: Given my admittedly limited understanding of how the LLMs that are behind these efforts work, and a certain amount of empirical observation, that the responses for any given prompt are… random but directed feels like as good a description as anything else. For example, I started two separate sessions with CodeGPT using the same prompt:

I would like you to generate some starting-point code, in Python, that will be used to keep track of a task-list for a locally-hosted web-application. The target audience is parents of children who have difficulties keeping track of their day-to-day obligations. The system should be able to keep track of those obligations (“chores”) and collections of “rewards” that they receive once their cores/obligations are complete.

Let’s start with a very high-level design for the backing logic for the application, concerned with representing the users, obligations, and rewards in the system, without being concerned yet about how those items’ data will be stored.

Despite the fact that the prompts were identical, the answers it came up with across those two sessions were significantly different, though there are at least some common elements between them. The class diagrams that were generated by that prompt in each session are shown in Figure 1.1, below.

Figure 1.1: The class diagrams from the same prompt in two different CodeGPT sessions

Both of these are viable starting points for the design of the application mentioned in the prompt. Both have significantly different approaches implied just from the class structures that the LLM came up with. At this level, with contexts established for each, progressing deeper into actual functionality would not be difficult, so long as the context doesn’t get lost by the LLM somewhere along the way. As soon as that starts happening, though, there will be an even greater need for the kind of software engineering expertise that the balance of this book is about. Without that, the random nature of each prompt will tend to overwrite already-established and vetted code structure, and an inexperienced user will, I suspect, eventually drown in the churn of changes produced (at scale) by the LLM.

The rest of this book will examine, in some detail, the skillsets that are, in my opinion, critical to the discipline of software engineering.

Chapter 2 will re-examine the differences between programming — just writing code — and software engineering — knowing what code to write, and why.

Chapter 3 will discuss the idea of the Software Development Life Cycle (SDLC), and the kinds of activities and outcomes expected by the phases of that structure.

Chapter 4 will examine the ideas of system modeling and how the varied aspects of it shape system design and development alike.

Chapter 5 will explore the current process methodologies for software development, the paradigms that are commonly in use as code is being written, and various practices that can be leveraged to improve the speed and quality of development outcomes.

Chapter 6 will present common, standard elements of code style, documentation, and commenting, and explore when, why, and how to apply them.

Chapter 7 will focus similar examination and decision-making around functional standards for code.

Chapter 8 will show options for several common tools that help manage common Python project concerns and tasks, and some of the more common tools that are actually used to write Python code.

Chapter 9 will mark the beginning of the development story that the balance of the book is concerned with telling: the continuation of the story told in the first edition, about an imaginary company that is starting the process of moving their existing system from a collection of locally hosted services to a cloud-resident web-API-based system.

Chapter 10 will start making some decisions, in the context of the story about the project, based on information and options presented in the previous chapters.

Chapter 11 will go through the process of re-examining the existing Business Object idea in the system that is going to be rewritten, and make a decision about how the system’s data is going to be modeled and structured.

Chapter 12 will take the decisions made in Chapter 11 and determine how system data in Business Objects will be persisted to a back-end data store, after examining the options available for data storage and the criteria that relate to making those kinds of decisions.

Chapter 13 will explore the code structures needed to implement the chosen data persistence mechanism, and how to manage database changes as the needs of those Business Objects change over time.

Chapter 14 will dive into the processes needed to test the Business Objects and their data persistence processes.

Chapter 15 will explore the options available for automating the integration and deployment of code as it is written and approved for promotion to the production system.

Chapter 16 will work through the implementation needed to facilitate local development of an API system, allowing software engineers to work locally with the same code that will be deployed later into the cloud.

Chapter 17 will implement a proof-of-concept level set of API endpoints, using the Business Objects, API requirements, and local development frameworks discussed in previous chapters.

Chapter 18 will walk through the processes needed to actually deploy a functional database and API to an AWS account, leveraging AWS Infrastructure as Code tools, and exploring alternatives when and where they are needed.