Index

A

• analog data
  • collecting / Collecting analog data
• Apache CLF
  • URL / Looking at the CLF
• Apache logs
  • URL / Getting a web server log
• Arduino
  • defining / Background briefing: Arduino basics
  • references / Background briefing: Arduino basics
  • shopping list, organizing / Organizing a shopping list
  • flexible and responsive, to fluid situation / Getting it right the first time
  • bulk data, collecting with / Collecting bulk data with the Arduino
• Arduino IDE
  • obtaining / Getting the Arduino IDE
  • URL / Getting the Arduino IDE
• Arduino programming language
  • mastering / Mastering the Arduino programming language
  • arithmetic operators, using / Using the arithmetic and comparison operators
  • comparison operators, using / Using the arithmetic and comparison operators
  • common processing statements, using / Using common processing statements
  • Edit-Compile-Test-Break cycle, defining / Hacking and the edit, download, test and break cycle
• Arduino sensor data feed
  • about / Simple Arduino sensor data feed

B

• Beautiful Soup
  • upgrading / Mission One – upgrade Beautiful Soup
  • reference / Mission One – upgrade Beautiful Soup
  • HTML page, obtaining / Getting an HTML page
  • HTML structure, navigating / Navigating the HTML structure
• blinking light
  • about / Seeing a better blinking light
• blocks of text
  • displaying / Displaying blocks of text
• bulk data
  • collecting, with Arduino / Collecting bulk data with the Arduino
• Busy Waiting
  • about / Seeing a better blinking light
• Busy Waiting design
  • about / Simple Arduino sensor data feed

C

• CLF
  • definition / Looking at the CLF
  • fields / Looking at the CLF
  • pattern / Looking at the CLF
• collected data
  • formatting / Formatting the collected data
• Common Log Format (CLF)
  • about / Understanding the variety of formats
• complex layouts
  • defining / Understanding tables and complex layouts
• content filter
  • writing / Writing a content filter
• cooked strings
  • versus raw strings / Finding a pattern in a file
• CSV output
  • emitting / Emitting CSV output

D

• data
  • scraping, from PDF files / Scraping data from PDF files
  • collecting, from serial port / Collecting data from the serial port
• data collection
  • controlling / Controlling data collection
• data modeling and analysis
  • with Python / Data modeling and analysis with Python
• design principles, for methods
  • about / Extending the resource manager
• digital output pins
  • starting with / Starting with the digital output pins
  • external LED, designing / Designing an external LED
  • working prototype, assembling / Assembling a working prototype
• diode
  • about / Designing an external LED
• document
  • text data, obtaining from / Getting text data from a document
• downloads
  • tracking / What are they downloading?

E

• EARTH character
  • about / Using some simple statements
• entities
  • URL / Starting with someone we know
  • about / Starting with someone we know
• Expandable Cloud Computing (EC2)
  • about / Getting a web server log

F

• filter
  • noise, reducing with / Reducing noise with a simple filter
• Flickr
  • URL, for photos / Background briefing – images and social media
• Fritzing application
  • URL / Getting the Arduino IDE
• Fritzing tool
  • URL / Getting the Arduino IDE
• ftplib
  • solution, for obtaining logs from server / Building a more complete solution

G

• gadgets
  • building / Building our own gadgets
• generator expressions
  • Eager / Using generator expressions
  • Lazy / Using generator expressions
• Google Cloud Platform
  • URL / Who's doing the talking?
• grid
  • exposing / Exposing the grid
• gzip compressed file
  • reading / Reading a gzip compressed file

H

• http.client
  • web services, accessing with / Accessing web services with urllib or http.client
• Hypertext Transfer Protocol (HTTP)
  • about / Accessing web services with urllib or http.client

I

• image processing
  • with PIL package / Background briefing – images and social media

J

• JavaScript Object Notation (JSON)
  • about / Navigating the HTML structure, Background briefing – web servers and logs

L

• language analysis
  • about / Deep Under Cover – NLTK and language analysis
• Latin America or Caribbean (LAC)
  • about / Processing whois queries
• linear model
  • creating / Creating a linear model

N

• Network Information Center (NIC)
  • about / Processing whois queries
• NLTK
  • about / Deep Under Cover – NLTK and language analysis
  • reference link / Deep Under Cover – NLTK and language analysis
• noise
  • reducing, with simple filter / Reducing noise with a simple filter
• numbers
  • defining / Crunching the numbers

O

• other upgrades
  • defining / Doing other upgrades

P

• page iterator
  • filtering / Filtering the page iterator
• PDF
  • URL / Background briefing–Portable Document Format
• PDF content
  • extracting / Extracting PDF content
  • generator expressions, using / Using generator expressions
  • generator functions, writing / Writing generator functions
  • bad data, filtering / Filtering bad data
  • context manager, writing / Writing a context manager
  • PDF parser resource manager, writing / Writing a PDF parser resource manager
  • resource manager, extending / Extending the resource manager
• PDF document
  • URL / Extracting PDF content
  • about / Extracting PDF content
• PDFDocument
  • about / Extracting PDF content
• PDF files
  • data, scraping from / Scraping data from PDF files
• PDF Miner
  • references / Extracting PDF content
• PDF Miner 3k
  • URL / Scraping data from PDF files
• pdf package
  • URL / Scraping data from PDF files
• PDFPageInterpreter
  • about / Extracting PDF content
• PDFParser
  • about / Extracting PDF content
• PDFResourceManager
  • about / Extracting PDF content
• Pillow
  • references / Doing other upgrades
• PIL package
  • used, for processing image / Background briefing – images and social media
• pip3.4 program
  • about / Background briefing – images and social media
• pip application
  • using / Mission One – upgrade Beautiful Soup
• ply package
  • references / Scraping data from PDF files
  • defining / Sidebar on the ply package
• Portable Document Format
  • defining / Background briefing–Portable Document Format
• PostScript
  • about / Background briefing–Portable Document Format
• PyPI
  • URL / Mission One – upgrade Beautiful Soup
• PySerial
  • URL / Getting a Python serial interface
• Python
  • URL / Doing a Python upgrade
  • about / Sidebar on the ply package
  • used, for running other programs / Using Python to run other programs
• Python 3 compatibility
  • URL / Mission One – upgrade Beautiful Soup
• Python Enhancement Proposals (PEP)
  • about / Background briefing on tools
  • URL / Background briefing on tools
• Python language
  • reviewing / Background briefing: review of the Python language
  • variables used, for saving results / Using variables to save results
  • strings, using / Using the sequence collections: strings
  • tuples and lists, using / Using other common sequences: tuples and lists
  • dictionary mapping, using / Using the dictionary mapping
  • data, comparing / Comparing data and using the logic operators
  • logic operators, using / Comparing data and using the logic operators
  • simple statements, using / Using some simple statements
  • if statement, using / Using compound statements for conditions: if
  • for statement, using / Using compound statements for repetition: for and while
  • while statement, using / Using compound statements for repetition: for and while
  • functions, defining / Defining functions
  • script files, creating / Creating script files
• Python serial interface
  • obtaining / Getting a Python serial interface

R

• raw data
  • reading / Reading and understanding the raw data
  • Physical Format / Reading and understanding the raw data
  • Logical Layout / Reading and understanding the raw data
  • Conceptual Content / Reading and understanding the raw data
  • gzip compressed file, reading / Reading a gzip compressed file
• raw strings
  • versus cooked strings / Finding a pattern in a file
• Read Eval Print Loop (REPL)
  • about / Background briefing: review of the Python language
• referrer URL
  • parsing / Trails of activity
• regular expression
  • writing, for parsing / Writing a regular expression for parsing
  • rules and patterns / Introducing some regular expression rules and patterns
  • pattern, searching in file / Finding a pattern in a file
  • suffix operators, using / Using regular expression suffix operators
  • characters, capturing by name / Capturing characters by name
  • CLF / Looking at the CLF
• Regular Expression Strings
  • about / Writing a regular expression for parsing
• remote files
  • reading / Reading remote files
• Representational State Transfer (REST)
  • about / Background briefing – web servers and logs
• resistors
  • adding, for LEDs / Solving problems adding an audible alarm
• RFC 3912
  • URL / Processing whois queries

S

• semiconductor
  • about / Designing an external LED
• Spinning
  • about / Seeing a better blinking light
• suffix operators
  • using / Using regular expression suffix operators

T

• tables
  • defining / Understanding tables and complex layouts
• text block recognition tweaks
  • creating / Making some text block recognition tweaks
• text data
  • obtaining, from document / Getting text data from a document
• toolkit
  • expanding / Mission to expand our toolkit
• tools
  • defining / Background briefing on tools
  • reasons / Background briefing on tools
  • features / Background briefing on tools
  • performance / Background briefing on tools
  • security / Background briefing on tools
  • housecleaning / Background briefing on tools
  • Python, upgrading / Doing a Python upgrade
  • pip, upgrading / Preliminary mission to upgrade pip
• Twitter API project, on PyPI
  • URL / Mission to expand our toolkit
• Twitter project from sixohsix
  • URL / Mission to expand our toolkit
• Twitter social network
  • user information, gathering / Who's doing the talking?
  • profile information, obtaining / Starting with someone we know
  • URL / Starting with someone we know
  • followers, searching / Finding our followers
  • conversation, examining / What do they seem to be talking about?
  • images being posted, gathering / What are they posting?

U

• urllib
  • web services, accessing with / Accessing web services with urllib or http.client

W

• weather forecasts
  • URL / Mission One – upgrade Beautiful Soup
• web server logs
  • overview / Background briefing – web servers and logs
  • formats / Understanding the variety of formats
  • obtaining / Getting a web server log
  • studying / Studying a log in more detail
  • obtaining, from server with ftplib / Getting logs from a server with ftplib
• web services
  • accessing, with urllib / Accessing web services with urllib or http.client
  • accessing, with http.client / Accessing web services with urllib or http.client
• Whois program
  • about / Who is this person?
  • URL / Who is this person?
  • reference link / Who is this person?
  • Python, using / Using Python to run other programs
  • whois queries, processing / Processing whois queries
  • request, decomposing / Breaking a request into stanzas and lines
  • stanza-finding algorithm / Alternate stanza-finding algorithm
  • bulk requests, creating / Making bulk requests
• World Wide Web (WWW)
  • about / Accessing web services with urllib or http.client

Y

• Yacc (Yet Another Compiler Compiler)
  • about / Sidebar on the ply package