This book presents Python as a necessary tool to optimize digital forensic analysis—written from an examiner's perspective. In the first two chapters, we introduce the basics of Python in preparation for the remainder of this book, where we will develop scripts to accomplish forensic tasks. While focused on the use of the language as a tool, we will also explore the advantages of Python and how they allow many individuals in the field to create solutions for complex forensic challenges. Like Monty Python, Python's namesake, the next 12 chapters aim to present something completely different.

In this fast-paced field, a scripting language provides flexible problem solving in an automated fashion, allowing the examiner additional time to investigate other artifacts that, due to time constraints, may not have been analyzed as thoroughly otherwise. Admittedly, Python may not always be the right tool to complete the task at hand, but it is an invaluable tool to add to anyone's DFIR arsenal. Should you undertake the task of mastering Python, it will more than pay off the time investment as you will increase your analysis capabilities many fold and greatly diversify your skill set. This chapter outlines the basics of Python, from Hello World to core scripting concepts.

This chapter will cover the following topics:

• An introduction to Python and healthy development practices
• Basic programming concepts
• Manipulating and storing objects in Python
• Creating simple conditionals, loops, and functions

Python is a powerful forensic tool. However, before deciding to develop a script, it is important to consider the type of analysis that's required and the project timeline. In the examples that follow, we will outline situations where Python is invaluable and, conversely, when it is not worth the development effort. Though rapid development makes it easy to deploy a solution in a tough situation, Python is not always the best tool to implement. If a tool exists that performs the task at hand, and is available, it may be the more appropriate method for analysis.

Python is a preferred programming language for forensics due to its ease of use, library support, detailed documentation, and interoperability among operating systems. There are two main types of programming languages: those that are interpreted and those that are compiled. Compiling code allows the programming language to be converted into machine language. This lower-level language is more efficient for the computer to interpret. Interpreted languages are not as fast as compiled languages at runtime, but do not require compilation, which can take some time. Because Python is an interpreted language, we can make modifications to our code and immediately run and view the results. With a compiled language, we would have to wait for our code to re-compile before viewing the effect of our modifications. For this reason, Python may not run as quickly as a compiled language, but allows for rapid prototyping.

An incident response case presents an excellent example of when to use Python in a real-life setting. For example, let's consider that a client calls, panicked, reporting a data breach and is unsure of how many files were exfiltrated over the past 24 hours from their file server. Once on site, you are instructed to perform the fastest count of files accessed in the past 24 hours as this count, and the list of compromised files, will determine the course of action.

Python fits this bill quite nicely here. Armed with just a laptop, you can open a text editor and begin writing a solution. Python can be built and designed without the need for a fancy editor or toolset. The build process of your script may look like this, with each step building upon the previous one:

1. Make the script read a single file's last accessed timestamp
2. Write a loop that steps through directories and subdirectories
3. Test each file to see if that timestamp is from the past 24 hours
4. If it has been accessed within 24 hours, then create a list of affected files to display file paths and access times

The process here would result in a script that recurses over the entire server and output files found with a last accessed time in the past 24 hours for manual review. This script will likely be approximately 20 lines of code and have required 10 minutes, or less, for an intermediate scripter to develop and validate—it is apparent this would be more efficient than manually reviewing timestamps on the filesystem.

Before deploying any developed code, it is imperative that you validate its capability first. As Python is not a compiled language, we can easily run the script after adding new lines of code to ensure we haven't broken anything. This approach is known as test-then-code, a method commonly used in script development. Any software, regardless of who wrote it, should be scrutinized and evaluated to ensure accuracy and precision. Validation ensures that the code is operating properly, and although more time-consuming, provides reliable results that are capable of withstanding the courtroom, an important aspect in forensics.

A situation where Python may not be the best tool is for general case analysis. If you are handed a hard drive and asked to find evidence without additional insight, then a pre-existing tool will be the better solution. Python is invaluable for targeted solutions, such as analyzing a given file type and creating a metadata report. Developing a custom all-in-one solution for a given filesystem requires too much time to create when other tools, both paid and free, exist that support such generic analysis.

Python is useful in pre-processing automation. If you find yourself repeating the same tasks for each piece of evidence, it may be worthwhile to develop a system that automates those steps. A great example of suites that perform such analysis is ManTech's analysis and triage system (mantaray: http://github.com/mantarayforensics), which leverages a series of tools to create general reports that can speed up analysis when there is no scope of what data may exist.

When considering whether to commit resources to develop Python scripts, either on the fly or for larger projects, it is important to consider what solutions already exist, the time available to create a solution, and the time saved through automation. Despite best intentions, the development of solutions can go on for much longer than initially conceived without a strong design plan.

The development cycle involves at least five steps:

• Identify
• Plan
• Program
• Validate
• Bugs

The first step is self-explanatory; before you develop, you must identify the problem that needs to be solved. Planning is perhaps the most crucial step in the development cycle:

Good planning will help later by decreasing the amount of code required and the number of bugs. Planning becomes even more vital during the learning process. A forensic programmer must begin to answer the following questions: how will data be ingested, what Python data types are most appropriate, are third-party libraries necessary, and how will the results be displayed to the examiner? In the beginning, just as if we were writing a term paper, it is a good idea to write, or draw, an outline of your program. As you become more proficient in Python, planning will become second nature, but initially, it is recommended to create an outline or write pseudocode.

Pseudocode is an informal way of writing code before filling in the details with actual code. Pseudocode can represent the bare bones of the program, such as defining pertinent variables and functions while describing how they will all fit together within the script's framework. Pseudocode for a function might look like this:

# open the database
  # read from the database using the sqlite3 library
  # store in variable called records
  for record in records: 
    # process database records here

After identifying and planning, the next three steps make up the largest part of the development cycle. Once your program has been sufficiently planned, it is time to start writing code! Once the code is written, break in your new program with as much test data as possible. Especially in forensics, it is critical to thoroughly test your code instead of relying on the results of one example. Without comprehensive debugging, the code can crash when it encounters something unexpected, or, even worse, it could provide the examiner with false information and lead them down the wrong path. After the code has been tested, it is time to release it and prepare for bug reports. We are not talking about insects here! Despite a programmer's best efforts, there will always be bugs in the code. Bugs have a nasty way of multiplying even as you squash one, perpetually causing the programming cycle to begin repeatedly.

Before we get started, it is necessary that you install Python on your machine. It is important to understand that, at the time of writing this book, there are two supported versions of Python: Python 2 and 3. We will use both Python 2 and 3 to develop our solutions. Historically, many of the useful third-party forensic libraries were developed for Python 2. At this point, most libraries are compatible with Python 3, which has superior Unicode handling, a major headache in Python 2, among a number of other improvements. All of the code in this book has been tested with the latest appropriate versions of Python 2 (v. 2.7.15) or 3 (v. 3.7.1). In some cases, our code is compatible with both Python 2 and 3, or only works with one of the two. Each chapter will describe what version of Python is required to run the code.

Additionally, we recommend using an integrated development environment, or IDE, such as JetBrain's PyCharm. An IDE will highlight errors and offer suggestions that help streamline the development process and promote best practices when writing code. In the case that the installation of an IDE is not available, a simple text editor will work. We recommend an application such as Notepad++, Sublime Text, or Visual Studio Code. For those who are command line orientated, an editor such as vim or nano will work as well.

With Python installed, let's open the interactive prompt by typing python into your Command Prompt or Terminal. We will begin by introducing some built-in functions for use in troubleshooting. The first line of defense when confused by any object or function discussed in this book, or found in the wild, is the type(), dir(), and help() built-in functions. We realize we have not yet introduced common data types and so the following code might appear confusing.

However, that is exactly the point of this exercise. During development, you will encounter data types you are unfamiliar with or be unsure what methods exist to interact with the object. These three functions help solve those issues. We will introduce the fundamental data types later in this chapter.

The type() function, when supplied with an object, will return its __name__ attribute, providing type identifying information about the object. The dir() function, when supplied with a string representing the name of an object, will return its attributes, showing the available options of the functions and parameters belonging to the object. The help() function can be used to display the specifics of these methods through its docstrings. Docstrings are nothing more than descriptions of a function that detail the inputs, outputs, and how to use the function.

Let's look at the str, or string, object as an example of these three functions. In the following example, passing a series of characters surrounded by single quotes to the type() function results in a type of str, or string.

Here is an example:

>>> type('what am I?')
<class 'str'> 

If we pass in an object to the dir() function, such as str, we can see its methods and attributes. Let's say that we want to know what one of these functions, title(), does. We can use the help() function specifying the object and its function as the input.

The output of the function tells us no input is required, the output is a string object, and that the function capitalized the first character of every word. Let's use the title method on the what am I? string:

>>> dir(str) 
['__add__', '__class__', '__contains__', '__delattr__',
'__doc__', '__eq__', 
...
'swapcase', 'title', 'translate', 'upper', 'zfill']

>>> help(str.title)
Help on method_descriptor:

title(...)
    S.title() -> str

    Return a titlecased version of S, i.e. words start with title case characters, all remaining cased characters have lower case.

>>> 'what am I?'.title()
'What Am I?' 

Next, type number = 5. Now we have created a variable, called number, that has the numerical value of 5. Using type() on that object indicates that 5 is an int, or integer. Going through the same procedure as before, we can see a series of available attributes and functions for the integer object. With the help() function, we can check what the __add__() function does for our number object. From the following output, we can see that this function is equivalent to using the + symbol on two values:

>>> number = 5
>>> type(number)
<class 'int'>

>>> dir(number)
>>> ['__abs__', '__add__', __and__', '__class__', '__cmp__', '__coerce__',
...
'denominator', 'imag', 'numerator', 'real']

>>> help(number.__add__)
__add__(...)
x.__add__(y) <==> x+y

Let's compare the difference between the __add__() function and the + symbol to verify our assumption. Using both methods to add 3 to our number object results in a returned value of 8, as expected. Unfortunately, we've also broken a best practice rule illustrating this example:

>>> number.__add__(3)
8
>>> number + 3
8

Notice how some methods, such as __add__(), have double leading and trailing underscores. These are referred to as magic methods, and are methods the Python interpreter calls and should not be called by the programmer. These magic methods are instead called indirectly by the user. For example, the integer __add__() magic method is called when using the + symbol between two numbers. Following the previous example, you should never run number.__add__(3) instead of number + 3.

Python, like any other programming language, has a specific syntax. Compared to other common programming languages, Python is rather English-like and can be read fairly easily in scripts. This feature has attracted many, including the forensics community, to use this language. Even though Python's language is easy to read, it is not to be underestimated as it is powerful and supports common programming paradigms.

Most programmers start with a simple Hello World script, a test that proves they are able to execute code and print the famous message into the console window. With Python, the code to print this statement is a single line, as seen here, written on the first line of a file:

001 print("Hello World!") 

Save this line of code in a file called hello.py. To run this script, we call Python and the name of the script. If you are using Python 3, the message Hello World! should be displayed in your Terminal:

Let's discuss why this simple script will not execute successfully in some versions of Python 2.

Printing in Python is a very common technique as it allows the developer to display text to the console as the script executes. While there are many differences between Python 2 and 3, the way printing is called is the most obvious change, and is the reason why our previous example primarily only works with Python 3 as it is currently written. With Python 3, print became a function rather than a statement, as was the case with older versions of Python 2. Let's revisit our previous script and see a slight difference.

Note the following for Python 3:

001 print("Hello World!") 

Note the following for Python 2:

001 print "Hello World!"

The difference is seemingly minor. In Python 2, where print is a statement, you do not need to wrap what is being printed in parentheses. It would be disingenuous to say the difference is just semantics; however, for now just understand that print is written in two different ways, depending on the version of Python being used. The ramifications of this minor change mean that legacy Python 2 scripts that use print as a statement cannot be executed by Python 3.

Where possible, our scripts will be written to be compatible with both versions of Python. This goal, while seemingly impossible due to the difference in print, can be accomplished by importing a special Python library, called __future__, and changing the print statement to a function. To do this, we need to import the print function from the __future__ library and then write all print commands as function.

The following script executes in both Python 2 and 3:

001 from __future__ import print_function
002 print("Hello World!") 

In the previous screenshot, you can see the result of this script in Python 2.7.15 and Python 3.7.1.

With our first script complete, it is time to understand the basic data types of Python. These data types are similar to those found in other programming languages, but are invoked with a simple syntax, which is described in the following table and sections. For a full list of standard data types available in Python, visit the official documentation at https://docs.python.org/3/library/stdtypes.html:

You will find that most of our scripts can be accomplished using only the standard data types Python offers. Before we take a look at one of the most common data types, strings, we will introduce comments.

Something that is always said, and can never be said enough, is to comment your code. In Python, comments are formed by any line beginning with the pound, or more recently known as the hashtag, # symbol. When Python encounters this symbol, it skips the remainder of the line and proceeds to the next line. For comments that span multiple lines, we can use three single or double quotes to mark the beginning and end of the comments rather than using a single pound symbol for every line. What follows are examples of types of comments in a file called comments.py. When running this script, we should only see 10 printed to the console as all comments are ignored:

# This is a comment
print(5 + 5) # This is an inline comment.
# Everything to the right of the # symbol
# does not get executed
"""We can use three quotes to create 
multi-line comments."""  

The output is as follows:

When this code is executed, we only see the preceding at the console.

Strings are a data type that contain any character, including alphanumeric characters, symbols, Unicode, and other codecs. With the vast amount of information that can be stored as a string, it is no surprise they are one of the most common data types. Examples of areas where strings are found include reading arguments at the command line, user input, data from files, and outputting data. To begin, let us look at how we can define a string in Python.

There are three ways to create a string: with single quotes, double quotes, or with the built-in str() constructor method. Note that there is no difference between single- and double-quoted strings. Having multiple ways to create a string is advantageous, as it allows us to differentiate between intentional quotes within a string. For example, in the 'I hate when people use "air-quotes"!' string, we use the single quotes to demarcate the beginning and end of the main string. The double quotes inside the string will not cause any issues with the Python interpreter. Let's verify with the type() function that both single and double quotes create the same type of object:

>>> type('Hello World!')
<class 'str'>
>>> type("Foo Bar 1234")
<class 'str'> 

As we saw with comments, a block string can be defined by three single or double quotes to create multi-line strings. The only difference is whether we do something with the block-quoted value or not:

>>> """This is also a string""" 
This is also a string
>>> '''it 
 can span 
 several lines''' 
it\ncan span\nseveral lines 

The \n character in the returned line signifies a line feed or a new line. The output in the interpreter displays these newline characters as \n, though when fed into a file or console, a new line is created. The \n character is one of the common escape characters in Python. Escape characters are denoted by a backslash following a specific character. Other common escape characters include \t for horizontal tabs, \r for carriage returns, \', \", and \\ for literal single quotes, double quotes, and backslashes, among others. Literal characters allow us to use these characters without unintentionally using their special meaning in Python's context.

We can also use the add (+) or multiply (*) operators with strings. The add operator is used to concatenate strings together, and the multiply operator will repeat the provided string values:

>>> 'Hello' + ' ' + 'World'
Hello World
>>> "Are we there yet? " * 3
Are we there yet? Are we there yet? Are we there yet?

Let's look at some common functions we use with strings. We can remove characters from the beginning or end of a string using the strip() function. The strip() function requires the character we want to remove as its input, otherwise it will replace whitespace by default. Similarly, the replace() function takes two inputs the character to replace and what to replace it with. The major difference between these two functions is that strip() only looks at the beginning and end of a string:

# This will remove colon (`:`) from the beginning and end of the line
>>> ':HelloWorld:'.strip(':')
HelloWorld


# This will remove the colon (`:`) from the line and place a 
# space (` `) in it's place
>>> 'Hello:World'.replace(':', ' ')
Hello World 

We can check if a character or characters are in a string using the in statement. Or, we can be more specific, and check if a string startswith() or endswith() a specific character(s) instead (you know a language is easy to understand when you can create sensible sentences out of functions). These methods return True or False Boolean objects:

>>> 'a' in 'Chapter 2'
True
>>> 'Chapter 1'.startswith('Chapter')
True
>>> 'Chapter 1'.endswith('1')
True 

We can quickly split a string into a list based on some delimiter. This can be helpful to quickly convert data separated by a delimiter into a list. For example, comma-separated values (CSV) data is separated by commas and could be split on that value:

>>> print("Hello, World!".split(','))
["Hello", " World!"] 

Formatting parameters can be used on strings to manipulate them and convert them based on provided values. With the .format() function, we can insert values into strings, pad numbers, and display patterns with simple formatting. This chapter will highlight a few examples of the .format() method, and we will introduce more complex features of it throughout this book. The .format() method replaces curly brackets with the provided values in order.

This is the most basic operation for inserting values into a string dynamically:

>>> "{} {} {} {}".format("Formatted", "strings", "are", "easy!")
'Formatted strings are easy!'

Our second example displays some of the expressions we can use to manipulate a string. Inside the curly brackets, we place a colon, which indicates that we are going to specify a format for interpretation. Following this colon, we specify that there should be at least six characters printed. If the supplied input is not six characters long, we prepend zeroes to the beginning of the input. Lastly, the d character specifies that the input will be a base 10 decimal:

>>> "{:06d}".format(42)
'000042' 

Our last example demonstrates how we can easily print a string of 20 equal signs by stating that our fill character is the equals symbol, followed by the caret (to center the symbols in the output), and the number of times to repeat the symbol. By providing this format string, we can quickly create visual separators in our outputs:

>>> "{:=^20}".format('')
'====================' 

The integer is another valuable data type that is frequently used—an integer is any whole positive or negative number. The float data type is similar, but allows us to use numbers requiring decimal-level precision. With integers and floats, we can use standard mathematical operations, such as: +, -, *, and /. These operations return slightly different results based on the object's type (for example, integer or float).

An integer uses whole numbers and rounding, for example dividing two integers will result in another whole number integer. However, by using one float in the equation, even one that has the same value as the integer will result in a float; for example, 3/2=1 and 3/2.0=1.5 in Python. The following are examples of integer and float operations:

>>> type(1010)
<class 'int'>
>>> 127*66
8382
>>> 66/10
6
>>> 10 * (10 - 8)
20 

We can use ** to raise an integer by a power. For example, in the following section, we raise 11 by the power of 2. In programming, it can be helpful to determine the numerator resulting from the division between two integers. For this, we use the modulus or percent (%) symbol. With Python, negative numbers are those with a dash character (-) preceding the value. We can use the built-in abs() function to get the absolute value of an integer or float:

>>> 11**2
121
>>> 11 % 2 # 11 divided by 2 is 5.5 or 5 with a remainder of 1
1
>>> abs(-3)
3

A float is defined by any number with a decimal. Floats follow the same rules and operations as we saw with integers, with the exception of the division behavior described previously:

>>> type(0.123)
<class 'float'>
>>> 1.23 * 5.23
6.4329
>>> 27/8.0
3.375

The integers 0 and 1 can also represent Boolean values in Python. These values are the Boolean False or True objects, respectively. To define a Boolean, we can use the bool() constructor statement. These data types are used extensively in program logic to evaluate statements for conditionals, as covered later in this chapter.

Another built-in data type is the null type, which is defined by the keyword None. When used, it represents an empty object, and when evaluated will return False. This is helpful when initializing a variable that may use several data types throughout execution. By assigning a null value, the variable remains sanitized until reassigned:

>>> bool(0)
False
>>> bool(1)
True
>>> None
>>> 

There are several data types that are more complex and allow us to create structures of raw data. This includes lists, dictionaries, sets, and tuples. Most of these structures are comprised of the previously mentioned data types. These structures are very useful in creating powerful units of values, allowing raw data to be stored in a manageable manner.

Lists are a series of ordered elements. Lists support any data type as an element and will maintain the order of data as it is appended to the list. Elements can be called by position or a loop can be used to step through each item. In Python, unlike other languages, printing a list takes one line. In languages like Java or C++, it can take three or more lines to print a list. Lists in Python can be as long as needed and can expand or contract on the fly, another feature uncommon in other languages.

We can create lists by using brackets with elements separated by commas. Or, we can use the list() class constructor with an iterable object. List elements can be accessed by index where 0 is the first element. To access an element by position, we place the desired index in brackets following the list object. Rather than needing to know how long a list is (which can be accomplished with the len() function), we can use negative index numbers to access list elements in reference to the end (that is, -3 would retrieve the third to last element):

>>> type(['element1', 2, 6.0, True, None, 234])
<class 'list'>
>>> list((4, 'element 2', None, False, .2))
[4, 'element 2', None, False, 0.2]
>>> len([0,1,2,3,4,5,6])
7
>>> ['hello_world', 'foo bar'][0]
hello_world
>>> ['hello_world', 'foo_bar'][-1]
foo_bar 

We can add, remove, or check if a value is in a list using a couple of different functions. The append() method adds data to the end of the list. Alternatively, the insert() method allows us to specify an index when adding data to the list. For example, we can add the string fish to the beginning, or 0 index, of our list:

>>> ['cat', 'dog'].append('fish')
# The list becomes: ['cat', 'dog', 'fish']
>>> ['cat', 'dog'].insert(0, 'fish')
# The list becomes: ['fish', 'cat', 'dog']  

The pop() and remove() functions delete data from a list either by index or by a specific object, respectively. If an index is not supplied with the pop() function, the last element in the list is popped. Note that the remove() function only gets rid of the first instance of the supplied object in the list:

>>> [0, 1, 2].pop()
2
# The list is now [0, 1]

>>> [3, 4, 5].pop(1)
4
# The list is now [3, 5]
>>> [1, 1, 2, 3].remove(1)
# The list becomes: [1, 2, 3] 

We can use the in statement to check if some object is in the list. The count() function tells us how many instances of an object are in the list:

>>> 'cat' in ['mountain lion', 'ox', 'cat']
True
>>> ['fish', 920.5, 3, 5, 3].count(3)
2 

If we want to access a subset of elements, we can use list slice notation. Other objects, such as strings, also support this same slice notation to obtain a subset of data. Slice notation has the following format, where a is our list or string object:

a[x:y:z]

In the preceding example, x represents the start of the slice, y represents the end of the slice, and z represents the step of the slice. Note that each segment is separated by colons and enclosed in square brackets. A negative step is a quick way to reverse the contents of an object that supports slice notation and would be triggered by a negative number as z. Each of these arguments is optional. In the first example, our slice returns the second element and up to, but not including, the fifth element in the list. Using just one of these slice elements returns a list containing everything from the second index forward or everything up to the fifth index:

>>> [0,1,2,3,4,5,6][2:5]
[2, 3, 4]
>>> [0,1,2,3,4,5,6][2:]
[2, 3, 4, 5, 6]
>>> [0,1,2,3,4,5,6][:5]
[0, 1, 2, 3, 4] 

Using the third slice element, we can skip every other element or simply reverse the list with a negative one. We can use a combination of these slice elements to specify how to carve a subset of data from the list:

>>> [0,1,2,3,4,5,6][::2]
[0, 2, 4, 6]
>>> [0,1,2,3,4,5,6][::-1]
[6, 5, 4, 3, 2, 1, 0]  

Dictionaries, otherwise known as dict, are another common Python data container. Unlike lists, this object does not add data in a linear fashion. Instead, data is stored as key and value pairs, where you can create and name unique keys to act as an index for stored values. It is important to note that, in Python 2, dictionaries do not preserve the order in which items are added to it. This is no longer true as of Python 3.6.5, though in general, we should not rely on the dict() object maintaining order for us. These objects are used heavily in forensic scripting, as they allow us to store data by name in a single object; otherwise, we may be left assigning a lot of new variables. By storing data in dictionaries, it is possible to have one variable contain very structured data.

We can define a dictionary by using curly braces ({}), where each key and value pair is delimited by a colon. Additionally, we can use the dict() class constructor to instantiate dictionary objects. Calling a value from a dictionary is accomplished by specifying the key in brackets following the dictionary object. If we supply a key that does not exist, we will receive a KeyError (notice that we have assigned our dictionary to a variable, a). While we have not introduced variables at this point, it is necessary to highlight some of the functions that are specific to dictionaries:

>>> type({'Key Lime Pie': 1, 'Blueberry Pie': 2})
<class 'dict'>
>>> dict((['key_1', 'value_1'],['key_2', 'value_2']))
{'key_1': 'value_1', 'key_2': 'value_2'}
>>> a = {'key1': 123, 'key2': 456}
>>> a['key1']
123 

We can add or modify the value of a preexisting key in a dictionary by specifying a key and setting it equal to another object. We can remove objects using the pop() function, similar to the list pop() function, to remove an item in a dictionary by specifying its key instead of an index:

>>> a['key3'] = 789
>>> a
{'key1': 123, 'key2': 456, 'key3': 789}
>>> a.pop('key1')
123
>>> a
{'key2': 456, 'key3': 789} 

The keys() and values() functions return a list of keys and values in the dictionary. We can use the items() function to return a list of tuples containing each key and value pair. These three functions are often used for conditionals and loops:

>>> a.keys()
dict_keys(['key2', 'key3'])
>>> a.values()
dict_values([456, 789])
>>> a.items()
dict_items([('key3', 789), ('key2', 456)])

Sets are similar to lists in that they contain a list of elements, though they must be unique items. With this, the elements must be immutable, meaning that the value must remain constant. For this, sets are best used on integers, strings, Boolean, floats, and tuples as elements. Sets do not index the elements, and therefore we cannot access the elements by their location in the set. Instead, we can access and remove elements through the use of the pop() method mentioned for the list method. Tuples are also similar to lists, though they are immutable. Built using parenthesis in lieu of brackets, elements do not have to be unique and of any data type:

>>> type(set([1, 4, 'asd', True]))
<class 'set'>
>>> g = set(["element1", "element2"])
>>> g
{'element1', 'element2'}
>>> g.pop()
'element2'
>>> g
{'element1'}
>>> tuple('foo')
('f', 'o' , 'o')
>>> ('b', 'a', 'r')
('b', 'a', 'r')
>>> ('Chapter1', 22)[0]
Chapter1
>>> ('Foo', 'Bar')[-1]
Bar 

The important difference between a tuple and a list is that a tuple is immutable. This means that we cannot change a tuple object. Instead, we must replace the object completely or cast it to a list, which is mutable. This casting process is described in the next section. Replacing an object is very slow since the operation to add a value to a tuple is tuple = tuple + ('New value',), noting that the trailing comma is required to denote that this addition is a tuple.

In some situations, the initial data type might not be the desired data type and needs to be changed while preserving its content. For example, when a user inputs arguments from the command line, they are commonly captured as strings and sometimes that user input needs to be, for example, an integer. We would need to use the integer class constructor to convert that string object before processing the data. Imagine we have a simple script that returns the square of a user-supplied integer; we would need to first convert the user input to an integer prior to squaring the input. One of the most common ways to convert data types is to wrap the variable or string with the constructor method, as shown here, for each of the data types:

>>> int('123456') # The string 123456
123456 # Is now the integer 123456
>>> str(45) # The integer 45
'45' # Is now the string 45
>>> float('37.5') # The string 37.5
37.5 # Is now the float 37.5 

Invalid conversions, for example, converting the letter 'a' to an integer, will raise a ValueError. This error will state that the specified value cannot be converted to the desired type. In this case, we would want to use the built-in ord() method, which converts a character to its integer equivalent based on the ASCII value. In other scenarios, we may need to use other methods to convert between data types. The following is a table of common built-in data type conversion methods we can utilize for most scenarios:

We can also interchange the type or ordered collections found in our list, set, and tuple types. Since sets have requirements for what data may be inserted, we generally do not cast anything to a set. It is more common, instead, to case a set to a list so that we can access values by position:

>>> tuple_1 = (0, 1, 2, 3, 3)
>>> tuple_1
(0, 1, 2, 3, 3)
>>> set_1 = set(tuple_1)
>>> set_1
{0, 1, 2, 3}
>>> list_1 = list(tuple_1)
>>> list_1
[0, 1, 2, 3, 3]
>>> list_2 = list(set_1)
>>> list_2
[0, 1, 2, 3]

We will often create file objects to read or write data from a file. File objects can be created using the built-in open() method. The open() function takes two arguments, the name of the file and the mode. These modes dictate how we can interact with the file object. The mode argument is optional, and if left blank defaults to read-only. The following table illustrates the different file modes available for use:

Most often, we will use read and write in standard or binary mode. Let's take a look at a few examples and some of the common functions we might use. For this section, we will create a text file called file.txt with the following content:

This is a simple test for file manipulation.
We will often find ourselves interacting with file objects.
It pays to get comfortable with these objects.

In the following example, we open a file object that exists, file.txt, and assign it to a variable, in_file. Since we do not supply a file mode, it is opened in read-only mode by default. We can use the read() method to read all lines as a continuous string. The readline() method can be used to read individual lines as a string. Alternatively, the readlines() method creates a string for each line and stores it in a list. These functions take an optional argument, specifying the size of bytes to read.

Python keeps track of where we currently are in the file. To illustrate the examples we've described, we need to use the seek() operation to bring us back to the start of the file before we run our next example. The seek() operation accepts a number and will navigate to that decimal character offset within the file. For example, if we tried to use the read() method before seeking back to the start, our next print function (showcasing the readline() method) would not return anything. This is because the cursor would be at the end of the file as a result of the read() function:

>>> in_file = open('file.txt')
>>> print(in_file.read())
This is a simple test for file manipulation.
We will often find ourselves interacting with file objects.
It pays to get comfortable with these objects.
>>> in_file.seek(0)
>>> print(in_file.readline())
This is a simple test for file manipulation.
>>> in_file.seek(0)
>>> print(in_file.readlines())
['This is a simple test for file manipulation.\n', 'We will often find ourselves interacting with file objects.\n', 'It pays to get comfortable with these objects.'] 

In a similar fashion, we can create, or open and overwrite, an existing file using the w file mode. We can use the write() function to write an individual string or the writelines() method to write any iterable object to the file. The writelines() function essentially calls the write() method for each element of the iterable object.

For example, this is tantamount to calling write() on each element of a list:

>>> out_file = open('output.txt', 'w')
>>> out_file.write('Hello output!')
>>> data = ['falken', 124, 'joshua']
>>> out_file.writelines(data) 

Python does a great job of closing connections to a file object automatically. However, best practice dictates that we should use the flush() and close() methods after we finish writing data to a file. The flush() method writes any data remaining in a buffer to the file, and the close() function closes our connection to the file object:

>>> out_file.flush()
>>> out_file.close() 

We can assign values to variables using the data types we just covered. By assigning values to variables, we can refer to that value, which could be a large 100-element list, by its variable name. This not only saves the programmer from re-typing out the value over and over again, but helps enhance the readability of the code and allows us to change the values of a variable over time. Throughout this chapter, we have already assigned objects to variables using the = sign. Variable names can technically be anything, although we recommend the following guidelines:

• Variable names should be short and descriptive of the stored content or purpose.
• Begin with a letter or underscore.
• Constant variables should be denoted by capitalized words.
• Dynamic variables should be lowercase words separated by underscores.
• Never be one of the following or any Python-reserved name: input, output, tmp, temp, in, for, next, file, True, False, None, str, int, list.
• Never include a space in a variable name. Python thinks two variables are being defined and will raise a syntax error. Use underscores to separate words.

Generally, programmers use memorable and descriptive names that indicate the data they hold. For example, in a script that prompts for the phone number of the user, the variable should be phone_number, which clearly indicates the purpose and contents of this variable. Another popular naming style is CamelCase, where every word is capitalized. This naming convention is often used in conjunction with class names (more on those later in this book).

A variable assignment allows the value to be modified as the script runs. The general rule of thumb is to assign a value to a variable if it will be used again. Let's practice by creating variables and assigning them data types we have just learned about. While this is simple, we recommend following along in the interactive prompt to get in the habit of assigning variables. In the first example here, we assign a string to a variable before printing the variable:

>>> print(hello_world)
Hello World! 

The second example introduces some new operators. First, we assign the integer, 5, to the variable, our_number. Then, we use the plus-gets (+=) as a built-in shorthand for our_number = our_number + 20. In addition to plus-gets, there is minus-gets (-=), multiply-gets (*=), and divide-gets (/=):

>>> our_number = 5
>>> our_number += 20
>>> print(our_number)
25 

In the following code block, we assign a series of variables before printing them. The data types used for our variables are string, integer, float, list, and Boolean, respectively:

>>> BOOK_TITLE = 'Learning Python for Forensics'
>>> edition = 2
>>> python2_version = 2.7.15
>>> python3_version = 3.7.1
>>> AUTHOR_NAMES = ['Preston Miller', 'Chapin Bryce']
>>> is_written_in_english = True
>>> print(BOOK_TITLE)
'Learning Python for Forensics'
>>> print(AUTHOR_NAMES)
['Preston Miller', 'Chapin Bryce']
>>> print(edition)
1
>>> print(python2_version)
2.7.15
>>> print(is_written_in_english)
True

Notice the BOOK_TITLE and AUTHOR_NAMES variables. When a variable is static, for instance, non-changing throughout the execution of a script, it is referred to as a constant variable. Unlike other programming languages, there is not a built-in method for protecting constants from being overwritten, so we use naming conventions to assist in reminding us not to replace the value. While some variables such as the edition of the book, language, or version of Python might change, the title and authors should be constants (we hope). If there is ever confusion when it comes to naming and styling conventions in Python, try running the following statement in an interpreter:

>>> import this  

As we saw previously, we can use the split() method on a string to convert it into a list. We can also convert a list into a string using the join() method. This method follows a string containing the desired common denominator and the list as its only argument. In the following example, we are taking list containing two strings and joining them into one string, where the elements are separated by a comma:

>>> print(', '.join(["Hello", "World!"]))
Hello, World!

Flow control logic allows us to create dynamic operations by specifying different routes of program execution based upon a series of circumstances. In any script worth its salt, some manner of flow control is present. For example, flow logic would be required to create a dynamic script that returns different results based on options selected by the user. In Python, there are two basic sets of flow logic: conditionals and loops.

Flow operators are frequently accompanied by flow logic. These operators can be strung together to create more complicated logic. The following table represents a truth table and illustrates the value of various flow operators based on the A or B variable Boolean state:

The logical AND and OR operators are the third and fourth columns in the table. Both A and B must be True for the AND operator to return True. Only one of the variables needs to be True for the OR operator to be True. The not operator simply switches the Boolean value of the variable to its opposite (for example, True becomes False and vice versa).

Mastering conditionals and loops will take our scripts to another level. At its core, flow logic relies on only two values, True or False. As noted earlier, in Python, these are represented by the Boolean True and False data types.

When a script hits a conditional, it's much like standing at a fork in the road. Depending on some factor, say a more promising horizon, you may decide to go east over west. Computer logic is less arbitrary in that if something is true the script proceeds one way, and if it is false then it will go another. These junctions are critical; if the program decides to go off the path we've developed for it, we'll be in serious trouble.

There are three statements that are used to form a conditional block: if, elif, and else. The conditional block refers to the conditional statements, their flow logic, and code. A conditional block starts with an if statement followed by flow logic, a colon, and indented line(s) of code. If the flow logic evaluates to True, then the indented code following the if statement will be executed. If it does not evaluate to True, the Python virtual machine (PVM) will skip those lines of code and go to the next line on the same level of indentation as the if statement. This is usually a corresponding elif (else-if) or else statement.

Indentation is very important in Python. It is used to demarcate code to be executed within a conditional statement or loop. A standard of four spaces for indentation is used in this book, though you may encounter code that uses a two-space indentation or uses tab characters. While all three of these practices are allowed in Python, four spaces are preferred and easier to read.

In a conditional block, once one of the statements evaluates to True, the code is executed and the PVM exits the block without evaluating the other statements.

# Conditional Block Pseudocode
if [logic]:
    # Line(s) of indented code to execute if logic evaluates to True.
elif [logic]:
    # Line(s) of indented code to execute if the 'if' 
    # statement is false and this logic is True.
else:
    # Line(s) of code to catch all other possibilities if
    # the 'if' and 'elif' statements are all False.

Until we define functions, we will stick to simple if statement examples:

>>> a = 5
>>> b = 22
>>> a > 0
True
>>> a > b
False
>>> if a > 0:
...     print(str(a) + ' is greater than zero!')
...
5 is greater than zero!
>>> if a >= b:
...     print(str(a) + ' beats ' + str(b))
...
>>> 

Notice how when the flow logic evaluates to True, then the code indented following the if statement is executed. When it evaluates to False, the code is skipped. Typically, when the if statement is false, you will have a secondary statement, such as an elif or else to catch other possibilities, such as when a is less than or equal to b. However, it is important to note that we can just use an if statement without any elif or else statements.

The difference between if and elif is subtle. We can only functionally notice a difference when we use multiple if statements. The elif statement allows for a second condition to be evaluated in the case that the first isn't successful. A second if statement will be evaluated regardless of the outcome of the first if statement.

The else statement does not require any flow logic and can be treated as a catch-all case for any remaining or unaccounted for case. This does not mean, however, errors will not occur when the code in the else statement is executed. Do not rely on else statements to handle errors.

Conditional statements can be made more comprehensive by using the logical and or or operators. These allow for more complex logic in a single conditional statement:

>>> a = 5
>>> b = 22
>>> if a > 4 and a < b:
...     print('Both statements must be true to print this')
...
Both statements must be true to print this
>>> if a > 10 or a < b:
...     print('One of these statements must be true to print this')
...
Only one of these statements must be true to print this 

The following table can be helpful to understand how common operators work:

Loops provide another method of flow control, and are suited to perform iterative tasks. A loop will repeat inclusive code until the provided condition is no longer True or an exit signal is provided. There are two kinds of loops: for and while. For most iterative tasks, a for loop will be the best option to use.

for loops are the most common and, in most cases, the preferred method to perform a task over and over again. Imagine a factory line; for each object on the conveyor belt, a for loop could be used to perform some task on it, such as placing a label on the object. In this manner, multiple for loops can come together in the form of an assembly line, processing each object, until they are ready to be presented to the user.

Much like the rest of Python, the for loop is very simple syntactically, yet powerful. In some languages, a for loop needs to be initialized, have a counter of sorts, and a termination case. Python's for loop is much more dynamic and handles these tasks on its own. These loops contain indented code that is executed line by line. If the object being iterated over still has elements (for example, more items to process) at the end of the indented block, the PVM will position itself at the beginning of the loop and repeat the code again.

The for loop syntax will specify the object to iterate over and what to call each of the elements within the object. Note that the object must be iterable. For example, lists, sets, tuples, and strings are iterable, but an integer is not. In the following example, we can see how a for loop treats strings and lists and helps us iterate over each element in iterable objects:

>>> for character in 'Python':
...      print(character)
...
P
y
t
h
o
n
>>> cars = ['Volkswagon', 'Audi', 'BMW']
>>> for car in cars:
...      print(car)
...
Volkswagon
Audi
BMW 

There are additional, more advanced, ways to call a for loop. The enumerate() function can be used to start an index. This comes in handy when you need to keep track of the index of the current loop. Indexes are incremented at the beginning of the loop. The first object has an index of 0, the second has an index of 1, and so on. The range() function can execute a loop a certain number of times and provide an index:

>>> numbers = [5, 25, 35]
>>> for i, x in enumerate(numbers):
...     print('Item', i, 'from the list is:', x)
...
Item 0 from the list is: 5
Item 1 from the list is: 25
Item 2 from the list is: 35
>>> for x in range(0, 100):
...     print(x)
0
1
# continues to print 0 to 100 (omitted in an effort to save trees)

while loops are not encountered as frequently in Python. A while loop executes as long as a statement is true. The simplest while loop would be a while True statement. This kind of loop would execute forever since the Boolean object True is always True and so the indented code would continually execute.

If you are not careful, you can inadvertently create an infinite loop, which will wreak havoc on your script's intended functionality. It is imperative to utilize conditionals to cover all your bases such as if, elif, and else statements. If you fail to do so, your script can enter an unaccounted situation and crash. This is not to say that while loops are not worth using. while loops are quite powerful and have their own place in Python:

>>> guess = 0
>>> answer = 42
>>> while True:
...     if guess == answer:
...          print('You've found the answer to this loop: ' + str(answer))
...          break
...     else:
...          print(guess, 'is not the answer.')
...          guess += 1 

The break, continue, and pass statements are used in conjunction with for and while loops to create more dynamic loops. The break escapes from the current loop, while the continue statement causes the PVM to begin executing code at the beginning of the loop, skipping any indented code following the continue statement. The pass statement literally does nothing and acts as a placeholder. If you're feeling brave or bored, or worse, both, remove the break statement from the previous example and note what happens.

Functions are the first step to creating more complex Python code. At a high level, they are containers of Python code that can be bundled together into a callable block. A simple model function requires a single input, performs an operation on the provided data, and returns a single output. However, this quickly becomes more complicated as functions can run without inputs or optional inputs or do not need to return an output at all.

Functions are an integral component of any programming language and have already been encountered many times in this chapter. For example, the append from list.append() is a function that requires input to add to a list. Once a function is created, you can invoke it by its name and pass any required inputs.

When it comes to writing functions, more is better. It is much easier to handle and troubleshoot a bug in a program with many small functions than one big function. Smaller functions make your code more readable and make it easier to find troublesome logic. That being said, functions should contain code for a singular purpose, such as accessing a certain key in a registry file. There is no need to create functions for each line of code in your script. Consider using functions as logical blocks of code. Sometimes that is three lines, sometimes that is 50 lines; what's important is that the purpose and operation of the functional unit of code is clear.

The function syntax starts with a definition, def, followed by the name of the function, any inputs in parenthesis, and a colon. Following this format are indented lines of code that will run when the function is called. Optionally, a function may have a return statement to pass information back to the instance where it was called from:

>>> def simple_function():
...      print('I am a simple function')
...
>>> simple_function()
I am a simple function 

In the example we've just seen, we've created a function named simple_function() that takes no inputs. This function does not return anything and instead prints a string. Let's take a look at more complicated examples.

Our first function, square(), takes one input and squares it. As this function returns a value, we catch it by assigning it to a variable when invoking the function. This variable, squared_number, will be equal to the returned value of the function. While this is a very succinct function, it is very easily broken if given the wrong input. Give the square function some other data type, such as a string, and you will receive a TypeError:

>>> def square(x):
...     return x**2
...
>>> squared_number = square(4)
>>> print(squared_number)
16

Our second function, even_or_odd, is slightly more advanced. This function first checks if it is passed an input that is of type integer. If not, it returns immediately, which causes the function to exit. If it is an integer, it performs some logic that displays to the user whether the integer is even or odd. Notice that when we try to give the function the string, '5', not to be confused with the integer, 5, it returns nothing, whereas in the square function, which lacks any input validation checks, this would have caused an error:

>>> def even_or_odd(value):
...     if isinstance(value, int):
...         if value % 2 == 0:
...               print('This number is even.')
...         else:
...              print('This number is odd.')
...      else:
...          return
...
>>> values = [1, 3, 4, 6, '5']
>>> for value in values:
...     even_or_odd(value)
...
This number is odd.
This number is odd.
This number is even.
This number is even.

Aspiring developers should get in the habit of writing functions. As always, functions should be well-commented to help explain their purpose. Functions will be used throughout this book, especially as we begin to develop our forensic scripts.

This chapter has covered a wide range of introductory content that provides a foundation to be built upon throughout the duration of this book; by the end, you will become well-versed in Python development. These topics have been handpicked as the most important items to comprise a basic understanding of the language as we move forward. We have covered data types, what they are and when they are used, variable naming and the associated rules and guidelines, logic and operations to manipulate and make decisions based on values, and conditions and loops that provide a sequential organization for our scripts and form the baseline of everything we develop. The code for this project can be downloaded from GitHub or Packt, as described in the Preface.

Please consider re-reading this chapter and working through the examples multiple times to help with comprehension. Just like anything else, learning a new language requires a lot of practice.

Through these features alone, we can create basic scripts. Python is a very powerful and complex language belying its simplistic syntax. In the next chapter, we will explore more complex foundational items and continue expanding upon knowledge established in this chapter, prior to moving on to real-world examples.