Getting logs from a server with ftplib

When we've created an analysis that HQ finds useful, we'll often have to scale this up to work on a larger supply of log files. This will involve acquiring and downloading files from servers without manually clicking a link to download and save each file.

We'll provide a sample of how we might use Python's ftplib to acquire files in bulk for analysis. Once we have the files locally, we can process them using our local_gzip() or local_text() functions.

Here's a function that performs a complex of FTP interaction:

import ftplib
def download( host, path, username=None ):
    with ftplib.FTP(host, timeout=10) as ftp:

        if username:
            password = getpass.getpass("Password: ")
            ftp.login(user=username,passwd=password)
        else:
            ftp.login()

        ftp.cwd(path)
        for name, facts in ftp.mlsd(".", ["type","size"]):
            if name.startswith("."): continue
            if facts['type'] == 'dir': continue
  ...