Secret Recipes of the Python Ninja

To illustrate scope, we will create nested functions, where a function is defined and then called within an enclosing function:

      >>> def first_funct():
      ...    x = 1
      ...    print(x)
      ...    def second_funct():
      ...        x = 2
      ...        print(x)
      ...    second_funct()
      ...

      >>> first_funct()
      1
      2

      >>> second_funct()
      Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      NameError: name 'second_funct' is not defined

      >>> import math

      >>> math.sin(45)
      0.8509035245341184

      >>> sin(45)
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
       NameError: name 'sin' is not defined

      >>> from math import *
      >>> sin(45)
      0.8509035245341184

        def print_funct(arg):
            print(arg)
            if __name__ == "__main__":
                import sys
                print_funct(sys.argv[1])

        def print_funct(arg1, arg2, arg3):
            print(arg1, arg2, arg3)
        if __name__ == "__main__":
            import sys
            print_funct(sys.argv[1], sys.argv[2], sys.argv[3])

      >>> def print_input(*args):
      ...   for val, input in enumerate(args):
      ...       print("{}. {}".format(val, input))
      ...
      >>> print_input("spam", "spam", "eggs", "spam")
      0. spam
      1. spam
      2. eggs
      3. spam

The location of a named assignment within the code determines its namespace visibility. In the preceding example, steps 1-3, if you directly call second_funct() immediately after calling first_funct(), you'll get an error stating second_funct() is not defined. This is true, because globally, the second function doesn't exist; it's nested within the first function and can't be seen outside the first function's scope. Everything within the first function is part of its namespace, just as the value for x within the second function can't be called directly but has to use the second_funct() call to get its value.

In the preceding examples, step 4-7, the math module is imported in its entirety, but it keeps its own namespace. Thus, calling math.sin() provides a result, but calling sin() by itself results in an error.

Then, the math module is imported using a wildcard. This tells the Python interpreter to import all the functions into the main namespace, rather than keeping them within the separate math namespace. This time, when sin() is called by itself, it provides the correct answer.

This demonstrates the point that namespaces are important to keep code separated while allowing the use of the same variables and function names. By using dot-nomenclature, the exact object can be called with no fear of name shadowing causing the wrong result to be provided.

In preceding examples, steps 7-10, using sys.argv() allows Python to parse command-line arguments and places them in a list for use. sys.argv([0]) is always the name of the program taking the arguments, so it can be safely ignored. All other arguments are stored in a list and can, therefore, be accessed by their index value.

Using *args tells Python to accept any number of arguments, allowing the program to accept a varying number of input values. An alternative version, **kwargs, does the same thing but with keyword:value pairs.

In addition to knowing about namespaces, there are some other important terms to know about when installing and working with modules:

The following is part of dice_roller.py, an example of embedded tests from one of the first Python programs this author wrote when first learning Python:

import random
def randomNumGen(choice):
    if choice == 1: #d6 roll
        die = random.randint(1, 6)
    elif choice == 2: #d10 roll
        die = random.randint(1, 10)
    elif choice == 3: #d100 roll
        die = random.randint(1, 100)
    elif choice == 4: #d4 roll
      die = random.randint(1, 4)
    elif choice == 5: #d8 roll
      die = random.randint(1, 8)
    elif choice == 6: #d12 roll
      die = random.randint(1, 12)
    elif choice == 7: #d20 roll
      die = random.randint(1, 20)
    else: #simple error message
        return "Shouldn't be here. Invalid choice"
    return die
if __name__ == "__main__":
    import sys
    print(randomNumGen(int(sys.argv[1])))

In this example, we are simply creating a random number generator that simulates rolling different polyhedral dice (commonly used in role-playing games). The random library is imported, then the function defining how the dice rolls are generated is created. For each die roll, the integer provided indicates how many sides the die has. With this method, any number of possible values can be simulated with a single integer input.

The key part of this program is at the end. The part if __name__ == "__main__" tells Python that, if the namespace for the module is main, that is, it is the main program and not imported into another program, then the interpreter should run the code below this line. Otherwise, when imported, only the code above this line is available to the main program. (It's also worth noting that this line is necessary for cross-platform compatibility with Windows.)

When this program is called from the command line, the sys library is imported. Then, the first argument provided to the program is read from the command line and passed into the randomNumGen() function as an argument. The result is printed to the screen. Following are some examples of results from this program:

$ python3 dice_roller.py 1
2
$ python3 dice_roller.py 2
10
$ python3 dice_roller.py 3
63
$ python3 dice_roller.py 4
2
$ python3 dice_roller.py 5
5
$ python3 dice_roller.py 6
6
$ python3 dice_roller.py 7
17
$ python3 dice_roller.py 8
Shouldn't be here. Invalid choice

Configuring a module in this manner is an easy way to allow a user to interface directly with the module on a stand-alone basis. It is also a great way to run tests on the script; the tests are only run when the file is called as a stand-alone, otherwise the tests are ignored. dice_roller_tests.py is the full dice-rolling simulator that this author wrote:

import random #randint
def randomNumGen(choice):
    """Get a random number to simulate a d6, d10, or d100 roll."""
    
    if choice == 1: #d6 roll
      die = random.randint(1, 6)
    elif choice == 2: #d10 roll
        die = random.randint(1, 10)
    elif choice == 3: #d100 roll
        die = random.randint(1, 100)
    elif choice == 4: #d4 roll
      die = random.randint(1, 4)
    elif choice == 5: #d8 roll
      die = random.randint(1, 8)
    elif choice == 6: #d12 roll
      die = random.randint(1, 12)
    elif choice == 7: #d20 roll
      die = random.randint(1, 20)
    else: #simple error message
        return "Shouldn't be here. Invalid choice"
    return die
def multiDie(dice_number, die_type):
    """Add die rolls together, e.g. 2d6, 4d10, etc."""
    
#---Initialize variables 
    final_roll = 0
    val = 0
    
    while val < dice_number:
        final_roll += randomNumGen(die_type)
        val += 1
    return final_roll
def test():
    """Test criteria to show script works."""
    
    _1d6 = multiDie(1,1) #1d6
    print("1d6 = ", _1d6, end=' ') 
    _2d6 = multiDie(2,1) #2d6
    print("\n2d6 = ", _2d6, end=' ')
    _3d6 = multiDie(3,1) #3d6
    print("\n3d6 = ", _3d6, end=' ')
    _4d6 = multiDie(4,1) #4d6
    print("\n4d6 = ", _4d6, end=' ')
    _1d10 = multiDie(1,2) #1d10
    print("\n1d10 = ", _1d10, end=' ')
    _2d10 = multiDie(2,2) #2d10
    print("\n2d10 = ", _2d10, end=' ')
    _3d10 = multiDie(2,2) #3d10
    print("\n3d10 = ", _3d10, end=' ')
    _d100 = multiDie(1,3) #d100
    print("\n1d100 = ", _d100, end=' ') 
    
if __name__ == "__main__": #run test() if calling as a separate program
    test()

This program builds on the previous random-dice program by allowing multiple dice to be added together. In addition, the test() function only runs when the program is called by itself to provide a sanity check of the code. The test function would probably be better if it wasn't in a function with the rest of the code, as it is still accessible when the module is imported, as shown below:

>>> import dice_roller_tests.py
>>> dice_roller_tests.test()
1d6 = 1 
2d6 = 8 
3d6 = 10 
4d6 = 12 
1d10 = 5 
2d10 = 8 
3d10 = 6 
1d100 = 26

So, if you have any code you don't want to be accessible when the module is imported, make sure to include it below the line, as it were.

The old way to manage virtual environments was with the venv tool. To install it, use the command sudo apt install python3-venv.

To manage virtual environments in a modern way, the pipenv module (https://docs.pipenv.org/) was developed; it automatically creates and manages virtual environments for projects, as well as adding and removing packages from Pipfile when you install/uninstall packages. It can be installed using pip install pipenv.

Pipfile is an alternative to requirements.txt, which is used to specify exact versions of modules to include in a program. Pipfile actually comprises two separate files: Pipfile and (optionally) Pipfile.lock. Pipfile is simply a listing of the source location of imported modules, the module names themselves (defaulting to the most recent version), and any development packages that are required. pipfile.py, below, is an example of a Pipfile from the Pipenv site (https://docs.pipenv.org/basics/#example-pipfile-pipfile-lock):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"


[dev-packages]
pytest = "*"

Pipfile.lock takes the Pipfile and sets actual version numbers to all the packages, as well as identifying specific hashes for those files. Hashed values are beneficial to minimize security risks; that is, if a particular module version has a vulnerability, its hash value allows it to be easily identified, rather than having to search by version name or some other method. pipfile_lock.py, below, is an example of a Pipfile.lock file from the Pipenv site (https://docs.pipenv.org/basics/#example-pipfile-pipfile-lock):

{
  "_meta": {
    "hash": {
      "sha256": "8d14434df45e0ef884d6c3f6e8048ba72335637a8631cc44792f52fd20b6f97a"
    },
    "host-environment-markers": {
      "implementation_name": "cpython",
      "implementation_version": "3.6.1",
      "os_name": "posix",
      "platform_machine": "x86_64",
      "platform_python_implementation": "CPython",
      "platform_release": "16.7.0",
      "platform_system": "Darwin",
      "platform_version": "Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64",
      "python_full_version": "3.6.1",
      "python_version": "3.6",
      "sys_platform": "darwin"
    },
    "pipfile-spec": 5,
    "requires": {},
    "sources": [
      {
        "name": "pypi",
        "url": "https://pypi.python.org/simple",
        "verify_ssl": true
      }
    ]
  },
  "default": {
    "certifi": {
      "hashes": [
        "sha256:54a07c09c586b0e4c619f02a5e94e36619da8e2b053e20f594348c0611803704",
        "sha256:40523d2efb60523e113b44602298f0960e900388cf3bb6043f645cf57ea9e3f5"
      ],
      "version": "==2017.7.27.1"
    },
    "chardet": {
      "hashes": [
         "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691",
         "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae"
      ],
      "version": "==3.0.4"
    },
***further entries truncated***

The preceding pipenv example shows the developer changing to the desired directory for the project, and then invoking pipenv to simultaneously create the virtual environment, activate it, and install the desired module.

In addition to creating the virtual environment, once you have created your Python program, you can run the program using pipenv as well:

>>> pipenv run python3 <program_name>.py

Doing this ensures all installed packages in the virtual environment are available to your program, thus reducing the likelihood of unexpected errors.

When launching a pipenv shell, a new virtual environment is created, with indications of where the environment is created in the file system. In this case, two environment executables are created, referencing both the Python 3.6 command and the default Python command. (Depending on the systems, these may actually reference different versions of Python. For example, the default Python command may call the Python 2.7 environment instead of Python 3.6.)

On a side note, the -m option indicates that Python is to run the module as a stand-alone script, that is, its contents will be ran within the __main__ namespace. Doing this means you don't have to know the full path to the module, as Python will look for the script in sys.path. In other words, for modules that you would normally import into another Python file can be run directly from the command line.

In the example of running pipenv, the command takes advantage of the fact that Python allows the -m option to run a module directly or allow it to be imported; in this case, pipenv imports venv to create the virtual environment as part of the creation process.

      $ pip install <package_name>

      $ pip install <package_name>==1.2.2

Here is an example of downgrading pygments from our earlier install in pipenv:

      $ pip install "<package_name> >= 1.1"

      $ pip install <some_package>

      $ pip install /local_files/SomePackage-1.2-py2.py3-none-any.whl

The wheel file name format breaks down to <package_name>-<version>-<language_version>-<abi_tag>-<platform_tag>.whl. The package name is the name of the module to be installed, followed by the version of this particular wheel file.

The language version refers to Python 2 or Python 3; it can be as specific as necessary, such as py27 (any Python 2.7.x version) or py3 (any Python 3.x.x version).

The ABI tag refers to the Application Binary Interface. In the past, the underlying C API (Application Programming Interface) that the Python interpreter relies on changed with every release, typically by adding API features rather than changing or removing existing APIs. The Windows OS is particularly affected, where each Python feature release creates a new name for the Python Window's DLL.

The ABI refers to Python's binary compatibility. While changes to Python structure definitions may not break API compatibility, ABI compatibility may be affected. Most ABI issues occur from changes in the in-memory structure layout.

Since version 3.2, a limited set of API features has been guaranteed to be stable for the ABI. Specifying an ABI tag allows the developer to specify which Python implementations a package is compatible with, for example, PyPy versus CPython. Generally speaking, this tag is set to none, implying there is no specific ABI requirement.

The platform tag specifies which OS and CPU the wheel package is designed to run. This is normally any, unless the wheel's developer had a particular reason to limit the package to a specific system type.

      FooProject == 1.2 --hash=sha256:<hash_digest>

      m1

      m2<1.7

     m3>=1.5, <=2.0

While the preceding screenshot shows some version specifier requirements, here is an example showing some of the different ways to specify module versions in requirements.txt:

        flask
        flask-pretty == 0.2.0
        flask-security <= 3.0
        flask-oauthlib >= 0.9.0, <= 0.9.4

In this example, module m1 is specified as a requirement, but the version number doesn't matter; in this case, pip will install the latest version. However, because of the bug in the latest version of m2, an earlier version is specified to be installed. Finally, m3 must be a version between 1.5 and 2.0 to satisfy the installation. Naturally, if one of these conditions can't be met, the installation will fail and the offending library and version numbers will be displayed for further troubleshooting.

It's worth noting that pip doesn't have true dependency resolution; it will simply install the first file specified. Thus, it is possible to have dependency conflicts or a sub-dependency that doesn't match the actual requirement. This is why a requirements file is useful, as it alleviates some dependency problems.

Verifying hashes also ensures that a package can't be changed without its version number changing as well, such as in an automated server deployment. This is an ideal situation for efficiency, as it eliminates the need for a private index server that maintains only approved packages.

In the preceding example, <vcs> is the version control system being used; it could be a local server or an online service such as, GitHub. <tag_name> is the version control tag used to identify this particular update to the control system.

If a required dependency was a top-level requirement for the project, then that particular line in the requirements file can simply be replaced. If it is a sub-dependency of another file, then the above command would be added as a new line.

Constraints files differ from requirements files in one key way: putting a package in the constraints file does not cause the package to be installed, whereas a requirements file will install all packages listed. Constraints files are simply requirements files that control which version of a package will be installed, but provide no control over the actual installation.

      $ pip list --outdated
      docutils (Current: 0.10 Latest: 0.11)
      Sphinx (Current: 1.2.1 Latest: 1.2.2)

While it is possible to update all outdated packages at once, this is not available within pip itself. There are two primary options: the first involves using sed, awk, or grep to walk through the list of packages, find the outdated packages, and update them. Alternatively, install the package pip-review to see outdated packages and update them. In addition, a number of other tools have been created by different developers, as well as instructions on how to do it yourself, so you should decide which works best for you.

      $ pip show sphinx
      Name: Sphinx
      Version: 1.7.2
      Summary: Python documentation generator
      Home-page: http://sphinx-doc.org/
      Author: Georg Brandl
      Author-email: georg@python.org
      License: BSD
      Location: /my/env/lib/python2.7/site-packages
      Requires: docutils, snowballstemmer, alabaster, Pygments, 
                imagesize, Jinja2, babel, six

      $ pip search peppercorn
      pepperedform    - Helpers for using peppercorn with formprocess.
      peppercorn      - A library for converting a token stream into [...]

When searching for packages, the query can be a package name or simply a word, as pip will find all packages with that string in the package name or in the package description. This is a useful way to locate a package if you know what you want to do but don't know the actual name of the package.

Packages installed with python setup.py install, and program wrappers that were installed using python setup.py develop, cannot be uninstalled via pip, as they do not provide metadata about which files were installed.

A number of other options are available for listing files, such as listing only non-global packages, beta versions of packages, outputting the list in columns, and other tools that may prove useful.

Additional information can be shown by using the --verbose option, as shown in the following screenshot:

The verbose option shows the same information as the default mode, but also includes such information as the classifier information that would found on the package's PyPI page. While this information could obviously be found simply by going to the PyPI site, if you are on a stand-alone computer or otherwise unable to connect to the internet, this can be useful when figuring out whether a package is supported by our current environment or when looking for similar packages within a particular topic.

To create an archive (from the official documentation: https://pip.pypa.io/en/latest/user_guide/#installation-bundles), perform the following:

      $ tempdir = $(mktemp -d /tmp/archive_dir)

      $ pip wheel -r requirements.txt --wheel-dir = $tempdir

      $ cwd = `pwd`

      $ (cd "$tempdir"; tar -cjvf "$cwd/<archive>.tar.bz2" *)

To install from an archive, do the following:

      $ tempdir=$(mktemp -d /tmp/wheelhouse-XXXXX)

      $ (cd $tempdir; tar -xvf /path/to/<archive>.tar.bz2)

      $ pip install --force-reinstall --ignore-installed --upgrade --no-index --no-deps $tempdir/*

In the first example (creating an archive), a temporary directory is first made, then the wheel is created using a requirements file and placed in the temporary directory. Next, the cwd variable is created and set equal to the present working directory (pwd). Finally, a combined command is issued, changing to the temporary directory, and creating an archive file in cwd of all the files in the temporary directory.

In the second example (installing from an archive), a temporary directory is created. Then, a combined command is given to change to that temporary directory and extract the files that make up the archive file. Then, using pip, the bundled files are used to install the Python program onto the computer in the temporary directory.

--force-reinstall will reinstall all packages when upgrading, even if they are already current. --ignore-installed forces a reinstall, ignoring whether the packages are already present. --upgrade upgrades all specified packages to the newest version available. --no-index ignores the package index and only looks at at URLs to parse for archives. --no-deps ensures that no package dependencies are installed.

-O removes assertstatements from the compiled code. These statements provide some debugging help when testing the program, but generally aren't required for production code.

-OO removes both assert and __doc__ strings for even more size reduction.

When source code (.py) is read by the Python interpreter, the bytecode is generated and stored in __pycache__ as <module_name>.<version>.pyc. The .pyc extension indicates that it is compiled Python code. This naming convention is what allows different versions of Python code to exist simultaneously on the system.

When source code is modified, Python will automatically check the date with the compiled version in cache and, if it's out of date, will automatically recompile the bytecode. However, a module that is loaded directly from the command line will not be stored in __pycache__ and is recompiled every time. In addition, if there is no source module, the cache can't be checked, that is, a bytecode-only package won't have a cache associated with it.

Because bytecode is platform-independent (due to being run through the platform's interpreter), Python code can be released either as .py source files or as .pyc bytecode. This is where bytecode-only packages come into play; to provide a bit of obfuscation and (subjective) security, Python programs can be released without the source code and only the pre-compiled .pyc files are provided. In this case, the compiled code is placed in the source directory rather than the source-code files.

In this example, the whole video package comprises two sub-packages, video formats and video effects, with video effects having several sub-packages of its own. Within each package, each .py file is a separate module. During module importation, Python looks for packages on sys.path.

The inclusion of the __init__.py files is necessary so Python will treat the directories as packages. This prevents directories with common names from shadowing Python modules further along the search path. They also allow calling modules as stand-alone programs via the -m option, when calling Python programs.

Initialization files are normally empty but can contain initialization code for the package. They can also contain an __all__ list, which is a Python list of modules that should be imported whenever from <package> import * is used.

The reason for __all__ is for the developer to explicitly indicate which files should be imported. This is to prevent excessive delay from importing all modules within a package that aren't necessarily needed for other developers. It also limits the chance of undesired side-effects when a module is inadvertently imported. The catch is, the developer needs to update the __all__ list every time the package is updated.

Relative imports are based on the name of the current module. As the main module for a program always has the name "__main__", any modules that will be the main module of an application must use absolute imports.

To be honest, it is generally safer to use absolute imports just to make sure you know exactly what you're importing; with most development environments nowadays providing suggestions for paths, it is just as easy to write out the auto-populated path as it is to use relative paths.

If __all__ is not defined in __init__.py, then import * only imports the modules within the specified package, not all sub-packages or their modules. For example, from video.formats import * only imports the video formats; the modules in the effects/ directory will not be included.

This is a best practice for Python programmers: as the Zen of Python (https://www.python.org/dev/peps/pep-0020/) states, explicit is better than implicit. Thus, importing a specific sub-module from a package is a good thing, whereas import * is frowned upon because of the possibility of variable name conflicts.

Packages have the __path__ attribute, which is rarely used. This attribute is a list that has the name of the directory where the package's __init__.py file is located. This location is accessed before the rest of the code for the file is run.

Modifying the package path affects future searches for modules and sub-packages within the package. This is useful when it is necessary to extend the number of modules found during a package search.

      from distutils.core import setup
      import py2exe
      setup(console=['hello.py'])

      cxfreeze <program>.py --target-dir=<directory>

Other options are available for this command, such as compressing the bytecode, setting an initialization script, or even excluding modules.

If more functionality is required, a distutils setup script can be created. The command cxfreeze-quickstart can be used to generate a simple setup script; the cx_Freeze documentation provides an example setup.py file (cxfreeze_setup.py):

      import sys
      from cx_Freeze import setup, Executable

      # Dependencies are automatically detected, but it might need fine tuning.
      build_exe_options = {"packages": ["os"], "excludes": ["tkinter"]}

      # GUI applications require a different base on Windows (the default is for 
      # console application).
      base = None
      if sys.platform == "win32":
          base = "Win32GUI"

      setup(  name = "guifoo",
              version = "0.1",
              description = "My GUI application!",
              options = {"build_exe": build_exe_options},
              executables = [Executable("guifoo.py", base=base)])

To run the setup script, run the command: python setup.py build. This will create the directory build/, which contains the subdirectory exe.xxx, where xxx is the platform-specific executable binary indicator:

      pyinstaller <program>.py

This generates the binary bundle in the dist/ subdirectory. Naturally, there many other options available when running this command:

      from distutils.core import setup
      from Cython.Build import cythonize

      setup(
          ext_modules = cythonize("helloworld.pyx")
      )

      $ python setup.py build_ext --inplace

      nuitka --recurse-all <program>.py

      nuitka --module <module>.py

      nuitka --module <package> --recurse-directory=<package>

Depending on a user's system configuration, you may need to provide the Microsoft Visual C runtime DLL. The py2exe documentation provides different files to choose from, depending on the version of Python you are working with.

In addition, py2exe does not create the installation builder, that is, installation wizard. While it may not be necessary for your application, Windows users generally expect a wizard to be available when running an .exe file. A number of free, open-source, and proprietary installation builders are available.

One benefit of building Mac binaries is that they are simple to pack for distribution; once the .app file is generated, right-click on the file and choose Create Archive. After that, your application is ready to be shipped out.

A common problem with cx_Freeze is that the program doesn't automatically detect a file that needs to be copied. This frequently occurs if you are dynamically importing modules into your program, for example, a plugin system.

Binaries created by cx_Freeze are generated for the OS it was run on; for instance, to create a Windows .exe file, cx_Freeze has to be used on a Windows computer. Thus, to make a truly cross-platform Python program that is distributed as executable binaries, you must have access to other operating systems. This can be alleviated by using virtual machines, cloud hosts, or simply purchasing the relevant systems.

When PyInstaller is run, it analyzes the supplied Python program and creates a <program>.spec file in the same folder as the Python program. In addition, the build/ subdirectory is placed in the same location.

The build/ directory contains log files and the working files used to actually create the binary. After the executable file is generated, a dist/ directory is placed in the same location as the Python program, and the binary is placed in the dist/ directory.

The executable file generated by Nuitka will have the .exe extension on all platforms. It is still usable on non-Windows OSes, but it is recommended to change the extension to a system-specific one to avoid confusion.

The binary files created with any of the commands previously shown require Python to be installed on the end system, as well as any C extension modules that are used.

Your project files need to be configured in the proper way so they are of use to other developers, and are listed properly on PyPI. The most important step of this process is setting up the setup.py file, which sits in the root of your project's directory.

setup.py contains configuration data for your project, particularly the setup() function, which defines the details of the project. It is also the command-line interface for running commands related to the packaging process.

A license (license.txt) should be included with the package. This file is important because, in some areas, a package without an explicit license cannot be legally used or distributed by anyone but the copyright holder. Including the license ensures both the creator and users are legally protected against copyright infringement issues.

A manifest file is also important if you need to package files that aren't automatically included in the source distribution. By default, the following files are included in the package when generated (known as the standard include set):

Any files that don't meet these criteria, such as a license file, need to be included in a MANIFEST.ini template file. The manifest template is a list of instructions on how to generate the actual manifest file that lists the exact files to include in the source distribution.

The manifest template can include or exclude any desired files; wildcards are available as well. For example, manifest_template.py from the distutils package shows one way to list files:

include *.txt
recursive-include examples *.txt *.py
prune examples/sample?/build

This example indicates that all .txt files in the root directory should be included, as well as all .txt and .py files in the examples/ subdirectory. In addition, all directories that match examples/sample?/build will be excluded from the package.

The manifest file is processed after the defaults above are considered, so if you want to exclude files from the standard include set, you can explicitly list them in the manifest. If, however, you want to completely ignore all defaults in the standard set, you can use the --no-defaults option to completely disable the standard set.

The order of commands in the manifest template is important. After the standard include set is processed, the template commands are processed in order. Once that is done, the final resulting command set is processed; all files to be pruned are removed. The resulting list of files is written to the manifest file for future reference; the manifest file is then used to build the source distribution archive.

It is important to note that the manifest template does not affect binary distributions, such as wheels. It is only for use in source-file packaging.

As mentioned previously, setup.py is a key file for the packaging process, and the setup() function is what enables the details of the project to be defined.

There are a number of arguments that can be provided to the setup() function, some of which will be covered in the following list. A good example of this is shown is the Listing Packages section:

There is actually a document on PEP 440 (https://www.python.org/dev/peps/pep-0440/) that indicates how to write your version numbers. versioning.py is an example of versioning a project:

      2.1.0.dev1# Development release      2.1.0a1# Alpha Release      2.1.0b1# Beta Release      2.1.0rc1# Release Candidate      2.1.0# Final Release      2.1.0.post1# Post Release      2018.04# Date based release
      19          # Serial release

      python setup.py bdist_wheel --universal

--universal should only be used when there are no C extensions in use and the Python code runs on both Python 2 and Python 3 without needing modifications, such as running 2to3.bdist_wheel signifies that the distribution is a binary one, as opposed to a source distribution. When used in conjunction with --universal, it does not check to ensure that it is being used correctly, so no warnings will be provided if the criteria are not met. The reason universal wheels shouldn't be used with C extensions is because pip prefers wheels over source distributions. Since an incorrect wheel will mostly likely prevent the C extension from being built, the extension won't be available for use.

      python setup.py bdist_wheel

bdist_wheel will identify the code and build a wheel that is compatible for any Python installation with the same major version number, that is, 2.x or 3.x.

Before uploading to the main PyPI site, there is a PyPI test site (https://testpypi.python.org/pypi) you can practice with. This allows developers the opportunity to ensure they know what they are doing with the entire building and uploading process, so they don't break anything on the main site. The test site is cleaned up on a semi-regular basis, so it shouldn't be relied on as a storage site while developing.

In addition, check the long and short descriptions in your setup.py to ensure they are valid. Certain directives and URLs are forbidden and stripped during uploading; this is one reason why it is good to test your project on the PyPI test site to see if there are any problems with your configuration.

Before uploading to PyPI, you need to create a user account. Once you have manually created an account on the web site, you can create a $HOME/.pypirc file to store your username and password. This file will be referenced when uploading so you won't have to manually enter it every time. However, be aware that your PyPI password is stored in plaintext, so if you are concerned about that you will have to manually provide it for every upload.

Once you have a created a PyPI account, you can upload your distributions to PyPI via twine; for new distributions, twine will automatically handle the registration of the project on the site. Install twine as normal using pip.

twine securely authenticates users to the PyPI database using HTTPS. The older way of uploading packages to PyPI was using python setup.py upload; this was insecure as the data was transferred via unencrypted HTTP, so your login credentials could be sniffed. With twine, connections are made through verified TLS to prevent credential theft.

This also allows a developer to pre-create distribution files, whereas setup.py upload only works with distributions that are created at the same time. Thus, using twine, a developer is able to test files prior to uploading them to PyPI, to ensure they work.

Finally, you can pre-sign your uploads with digital signatures and attach the .asc certification files to the twine upload. This ensures the developer's password is entered into GPG and not some other software, such as malware.