Python Programming Blueprints

This application is intended to be flexible and allow developers to create different parsers for different weather websites. We are going to create a parser loader that will dynamically discover files inside of the parsers directory, load them, and make them available to be used by the application without requiring changes to any other parts of the code. Here are the rules that our loader will require when implementing new parsers:

With that said, let's go ahead and create the parser loader. Create a file namedparser_loader.py inside of the weatherterm/core directory and add the following content:

import os
import re
import inspect


def _get_parser_list(dirname):
    files = [f.replace('.py', '')
             for f in os.listdir(dirname)
             if not f.startswith('__')]

    return files


def _import_parsers(parserfiles):

    m = re.compile('.+parser$', re.I)

    _modules = __import__('weatherterm.parsers',
                          globals(),
                          locals(),
                          parserfiles,
                          0)

    _parsers = [(k, v) for k, v in inspect.getmembers(_modules)
                if inspect.ismodule(v) and m.match(k)]

    _classes = dict()

    for k, v in _parsers:
        _classes.update({k: v for k, v in inspect.getmembers(v)
                         if inspect.isclass(v) and m.match(k)})

    return _classes


def load(dirname):
    parserfiles = _get_parser_list(dirname)
    return _import_parsers(parserfiles)

First, the _get_parser_list function is executed and returns a list of all files located in weatherterm/parsers; it will filter the files based on the rules of the parser described previously. After returning a list of files, it is time to import the module. This is done by the _import_parsers function, which first imports the weatherterm.parsers module and makes use of the inspect package in the standard library to find the parser classes within the module.

The inspect.getmembers function returns a list of tuples where the first item is a key representing a property in the module, and the second item is the value, which can be of any type. In our scenario, we are interested in a property with a key ending with parser and with the value of type class.

Assuming that we already have a parser in place in the weatherterm/parsers  directory, the value returned by the inspect.getmembers(_modules) will look something like this:

[('WeatherComParser',
  <class 'weatherterm.parsers.weather_com_parser.WeatherComParser'>),
  ...]

Lastly, we loop through the items in the module and extract the parser classes, returning a dictionary containing the name of the class and the class object that will be later used to create instances of the parser.

Let's start creating the model that will represent all the information that our application will scrape from the weather website. The first item we are going to add is an enumeration to represent each option of the weather forecast we will provide to the users of our application. Create a file named forecast_type.py in the directory weatherterm/core with the following contents:

from enum import Enum, unique


@unique
class ForecastType(Enum):
    TODAY = 'today'
    FIVEDAYS = '5day'
    TENDAYS = '10day'
    WEEKEND = 'weekend'

Enumerations have been in Python's standard library since version 3.4 and they can be created using the syntax for creating classes. Just create a class inheriting from enum.Enum containing a set of unique properties set to constant values. Here, we have values for the four types of forecast that the application will provide, and where values such as ForecastType.TODAY, ForecastType.WEEKEND, and so on can be accessed.

Note that we are assigning constant values that are different from the property item of the enumeration, the reason being that later these values will be used to build the URL to make requests to the weather website.

The application needs one more enumeration to represent the temperature units that the user will be able to choose from in the command line. This enumeration will contain Celsius and Fahrenheit items. 

First, let's include a base enumeration. Create a file called base_enum.py in the weatherterm/core directory with the following contents:

from enum import Enum


class BaseEnum(Enum):
    def _generate_next_value_(name, start, count, last_value):
        return name

 BaseEnum is a very simple class inheriting from Enum . The only thing we want to do here is override the method _generate_next_value_ so that every enumeration that inherits from BaseEnum and has properties with the value set to auto()  will automatically get the same value as the property name.

Now, we can create an enumeration for the temperature units. Create a file called unit.py in the weatherterm/core directory with the following content:

from enum import auto, unique

from .base_enum import BaseEnum


@unique
class Unit(BaseEnum):
    CELSIUS = auto()
    FAHRENHEIT = auto()

This class inherits from the BaseEnum that we just created, and every property is set to auto(), meaning the value for every item in the enumeration will be set automatically for us. Since the Unit class inherits from BaseEnum, every time the auto() is called, the _generate_next_value_ method on BaseEnum will be invoked and will return the name of the property itself.

Before we try this out, let's create a file called __init__.py in the weatherterm/core directory and import the enumeration that we just created, like so:

from .unit import Unit

If we load this class in the Python REPL and check the values, the following will occur:

Python 3.6.2 (default, Sep 11 2017, 22:31:28) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from weatherterm.core import Unit
>>> [value for key, value in Unit.__members__.items()]
[<Unit.CELSIUS: 'CELSIUS'>, <Unit.FAHRENHEIT: 'FAHRENHEIT'>]

Another item that we also want to add to the core module of our application is a class to represent the weather forecast data that the parser returns. Let's go ahead and create a file named forecast.py in the weatherterm/core directory with the following contents:

from datetime import date

from .forecast_type import ForecastType


class Forecast:
    def __init__(
            self,
            current_temp,
            humidity,
            wind,
            high_temp=None,
            low_temp=None,
            description='',
            forecast_date=None,
            forecast_type=ForecastType.TODAY):
        self._current_temp = current_temp
        self._high_temp = high_temp
        self._low_temp = low_temp
        self._humidity = humidity
        self._wind = wind
        self._description = description
        self._forecast_type = forecast_type

        if forecast_date is None:
            self.forecast_date = date.today()
        else:
            self._forecast_date = forecast_date

    @property
    def forecast_date(self):
        return self._forecast_date

    @forecast_date.setter
    def forecast_date(self, forecast_date):
        self._forecast_date = forecast_date.strftime("%a %b %d")

    @property
    def current_temp(self):
        return self._current_temp

    @property
    def humidity(self):
        return self._humidity

    @property
    def wind(self):
        return self._wind

    @property
    def description(self):
        return self._description

    def __str__(self):
        temperature = None
        offset = ' ' * 4

        if self._forecast_type == ForecastType.TODAY:
            temperature = (f'{offset}{self._current_temp}\xb0\n'
                           f'{offset}High {self._high_temp}\xb0 / '
                           f'Low {self._low_temp}\xb0 ')
        else:
            temperature = (f'{offset}High {self._high_temp}\xb0 / '
                           f'Low {self._low_temp}\xb0 ')

        return(f'>> {self.forecast_date}\n'
               f'{temperature}'
               f'({self._description})\n'
               f'{offset}Wind: '
               f'{self._wind} / Humidity: {self._humidity}\n')

In the Forecast class, we will define properties for all the data we are going to parse: 

We can also implement two methods called forecast_date  with the decorators @property  and @forecast_date.setter . The @property  decorator will turn the method into a getter for the _forecast_date property of the Forecast class, and the @forecast_date.setter will turn the method into a setter.  The setter was defined here because, every time we need to set the date in an instance of Forecast, we need to make sure that it will be formatted accordingly. In the setter, we call the strftime method, passing the format codes %a (weekday abbreviated name), %b (monthly abbreviated name), and %d (day of the month).

Lastly, we override the __str__ method to allow us to format the output the way we would like when using the print, format, and str functions.

By default, the temperature unit used by weather.com is Fahrenheit, and we want to give the users of our application the option to use Celsius instead. So, let's go ahead and create one more file in the weatherterm/core directory called unit_converter.py with the following content:

from .unit import Unit


class UnitConverter:
    def __init__(self, parser_default_unit, dest_unit=None):
        self._parser_default_unit = parser_default_unit
        self.dest_unit = dest_unit

        self._convert_functions = {
            Unit.CELSIUS: self._to_celsius,
            Unit.FAHRENHEIT: self._to_fahrenheit,
        }

    @property
def dest_unit(self):
        return self._dest_unit

    @dest_unit.setter
    def dest_unit(self, dest_unit):
        self._dest_unit = dest_unit

    def convert(self, temp):

        try:
            temperature = float(temp)
        except ValueError:
            return 0

        if (self.dest_unit == self._parser_default_unit or
                self.dest_unit is None):
            return self._format_results(temperature)

        func = self._convert_functions[self.dest_unit]
        result = func(temperature)

        return self._format_results(result)

    def _format_results(self, value):
        return int(value) if value.is_integer() else f'{value:.1f}'

    def _to_celsius(self, fahrenheit_temp):
        result = (fahrenheit_temp - 32) * 5/9
        return result

    def _to_fahrenheit(self, celsius_temp):
        result = (celsius_temp * 9/5) + 32
        return result

This is the class that is going to make the temperature conversions from Celsius to Fahrenheit and vice versa. The initializer of this class gets two arguments; the default unit used by the parser and the destination unit. In the initializer, we will define a dictionary containing the functions that will be used for temperature unit conversion.

The convert method only gets one argument, the temperature. Here, the temperature is a string, so the first thing we need to do is try converting it to a float value; if it fails, it will return a zero value right away.

You can also verify whether the destination unit is the same as the parser's default unit or not. In that case, we don't need to continue and perform any conversion; we simply format the value and return it.

If we need to perform a conversion, we can look up the _convert_functions  dictionary to find the conversion function that we need to run. If we find the function we are looking for, we invoke it and return the formatted value.

The code snippet below shows the _format_results method, which is a utility method that will format the temperature value for us:

return int(value) if value.is_integer() else f'{value:.1f}'

The _format_results method checks if the number is an integer; the value.is_integer() will return True if the number is, for example, 10.0. If True, we will use the int function to convert the value to 10; otherwise, the value is returned as a fixed-point number with a precision of 1. The default precision in Python is 6. Lastly, there are two utility methods that perform the temperature conversions, _to_celsius and _to_fahrenheit.

Now, we only need to edit the __init__.py file in the weatherterm/core directory and include the following import statements:

from .base_enum import BaseEnum
from .unit_converter import UnitConverter
from .forecast_type import ForecastType
from .forecast import Forecast

We are going to add a class named Request that will be responsible for getting the data from the weather website. Let's add a file named request.py in the weatherterm/core directory with the following content:

import os
from selenium import webdriver


class Request:
    def __init__(self, base_url):
        self._phantomjs_path = os.path.join(os.curdir,
                                          'phantomjs/bin/phantomjs')
        self._base_url = base_url
        self._driver = webdriver.PhantomJS(self._phantomjs_path)

    def fetch_data(self, forecast, area):
        url = self._base_url.format(forecast=forecast, area=area)
        self._driver.get(url)

        if self._driver.title == '404 Not Found':
            error_message = ('Could not find the area that you '
                             'searching for')
            raise Exception(error_message)

        return self._driver.page_source

This class is very simple; the initializer defines the base URL and creates a PhantomJS driver, using the path where PhantomJS is installed. The fetch_data method formats the URL, adding the forecast option and the area. After that, the webdriver performs a request and returns the page source. If the title of the markup returned is 404 Not Found, it will raise an exception. Unfortunately, Selenium doesn't provide a proper way of getting the HTTP Status code; this would have been much better than comparing strings.

Now, we can import it in the __init__.py  file in the weatherterm/core directory:

from .request import Request

Now we have a parser loader to load any parser that we drop into the directory weatherterm/parsers,  we have a class representing the forecast model, and an enumeration ForecastType so we can specify which type of forecast we are parsing. The enumeration represents temperature units and utility functions to convert temperatures from Fahrenheit to Celsius and Celsius to Fahrenheit. So now, we should be ready to create the application's entry point to receive all the arguments passed by the user, run the parser, and present the data on the terminal.

The core functionality is in place and the entry point of the application with the argument parser will give the users of our application a much better experience. Now, it is finally the time we all have been waiting for, the time to start implementing the parser. We will start implementing the method to get today's weather forecast.

Since I am in Sweden, I will use the area code SWXX2372:1:SW (Stockholm, Sweden); however, you can use any area code you want. To get the area code of your choice, go to https://weather.com and search for the area you want. After selecting the area, the weather forecast for the current day will be displayed. Note that the URL changes, for example, when searching Stockholm, Sweden, the URL changes to:

https://weather.com/weather/today/l/SWXX2372:1:SW

For São Paulo, Brazil it will be:

https://weather.com/weather/today/l/BRXX0232:1:BR

Note that there is only one part of the URL that changes, and this is the area code that we want to pass as an argument to our application.

import re

from weatherterm.core import Forecast
from weatherterm.core import Request
from weatherterm.core import Unit
from weatherterm.core import UnitConverter

self._base_url = 'http://weather.com/weather/{forecast}/l/{area}'
self._request = Request(self._base_url)

self._temp_regex = re.compile('([0-9]+)\D{,2}([0-9]+)')
self._only_digits_regex = re.compile('[0-9]+')

self._unit_converter = UnitConverter(Unit.FAHRENHEIT)

def _get_data(self, container, search_items):
    scraped_data = {}

    for key, value in search_items.items():
        result = container.find(value, class_=key)

        data = None if result is None else result.get_text()

        if data is not None:
            scraped_data[key] = data

    return scraped_data

def _parse(self, container, criteria):
    results = [self._get_data(item, criteria)
               for item in container.children]

    return [result for result in results if result]

def _clear_str_number(self, str_number):
    result = self._only_digits_regex.match(str_number)
    return '--' if result is None else result.group()

def _get_additional_info(self, content):
    data = tuple(item.td.span.get_text()
                 for item in content.table.tbody.children)
    return data[:2]

Implementing today's weather forecast

It's time to start adding the implementation of the _today_forecast method, but first, we need to import BeautifulSoup. At the top of the file, add the following import statement:

from bs4 import BeautifulSoup

Now, we can start adding the _today_forecast method:

def _today_forecast(self, args):
    criteria = {
        'today_nowcard-temp': 'div',
        'today_nowcard-phrase': 'div',
        'today_nowcard-hilo': 'div',
        }

    content = self._request.fetch_data(args.forecast_option.value,
                                       args.area_code)

    bs = BeautifulSoup(content, 'html.parser')

    container = bs.find('section', class_='today_nowcard-container')

    weather_conditions = self._parse(container, criteria)

    if len(weather_conditions) < 1:
        raise Exception('Could not parse weather foreecast for 
        today.')

    weatherinfo = weather_conditions[0]

    temp_regex = re.compile(('H\s+(\d+|\-{,2}).+'
                             'L\s+(\d+|\-{,2})'))
    temp_info = temp_regex.search(weatherinfo['today_nowcard-hilo'])
    high_temp, low_temp = temp_info.groups()

    side = container.find('div', class_='today_nowcard-sidecar')
    humidity, wind = self._get_additional_info(side)

    curr_temp = self._clear_str_number(weatherinfo['today_nowcard- 
    temp'])

    self._unit_converter.dest_unit = args.unit

    td_forecast = Forecast(self._unit_converter.convert(curr_temp),
                           humidity,
                           wind,
                           high_temp=self._unit_converter.convert(
                               high_temp),
                           low_temp=self._unit_converter.convert(
                               low_temp),
                           description=weatherinfo['today_nowcard-
                            phrase'])

    return [td_forecast]

That is the function that will be called when the -td or --today flag is used on the command line. Let's break down this code so that we can easily understand what it does. Understanding this method is important because these methods parse data from other weather forecast options (five days, ten days, and weekend) that are very similar to this one.

The method's signature is quite simple; it only gets args, which is the Argument object that is created in the __main__ method. The first thing we do in this method is to create a criteria dictionary with all the DOM elements that we want to find in the markup:

criteria = {
    'today_nowcard-temp': 'div',
    'today_nowcard-phrase': 'div',
    'today_nowcard-hilo': 'div',
}

As mentioned before, the key to the criteria dictionary is the name of the DOM element's CSS class, and the value is the type of the HTML element:

• The today_nowcard-temp class is a CSS class of the DOM element containing the current temperature
• The today_nowcard-phrase class is a CSS class of the DOM element containing weather conditions text (Cloudy, Sunny, and so on)
• The today_nowcard-hilo class is the CSS class of the DOM element containing the highest and lowest temperature

Next, we are going to fetch, create, and use BeautifulSoup to parse the DOM:

content = self._request.fetch_data(args.forecast_option.value, 
                                   args.area_code)

bs = BeautifulSoup(content, 'html.parser')

container = bs.find('section', class_='today_nowcard-container')

weather_conditions = self._parse(container, criteria)

if len(weather_conditions) < 1:
    raise Exception('Could not parse weather forecast for today.')

weatherinfo = weather_conditions[0]

First, we make use of the fetch_data method of the Request class that we created on the core module and pass two arguments; the first is the forecast option and the second argument is the area code that we passed on the command line. After fetching the data, we create a BeautifulSoup object passing the content and a parser. Since we are getting back HTML, we use html.parser.

Now is the time to start looking for the HTML elements that we are interested in. Remember, we need to find an element that will be a container, and the _parser function will search through the children elements and try to find items that we defined in the dictionary criteria. For today's weather forecast, the element that contains all the data we need is a section element with the today_nowcard-container CSS class.

BeautifulSoup contains the find method, which we can use to find elements in the HTML DOM with specific criteria. Note that the keyword argument is called class_ and not class because class is a reserved word in Python.

Now that we have the container element, we can pass it to the _parse method, which will return a list. We perform a check if the result list contains at least one element and raise an exception if it is empty. If it is not empty, we just get the first element and assign it to the weatherinfo variable. The weatherinfo variable now contains a dictionary with all the items that we were looking for.

The next step is split the highest and lowest temperature:

temp_regex = re.compile(('H\s+(\d+|\-{,2}).+'
                         'L\s+(\d+|\-{,2})'))
temp_info = temp_regex.search(weatherinfo['today_nowcard-hilo'])
high_temp, low_temp = temp_info.groups()

We want to parse the text that has been extracted from the DOM element with the today_nowcard-hilo CSS class, and the text should look something like H 50 L 60, H -- L 60, and so on. An easy and simple way of extracting the text we want is to use a regular expression:

H\s+(\d+|\-{,2}).L\s+(\d+|\-{,2})

We can break this regular expression into two parts. First, we want to get the highest temperature—H\s+(\d+|\-{,2}); this means that it will match an H followed by some spaces, and then it will group a value that matches either numbers or a maximum of two dash symbols. After that, it will match any character. Lastly, comes the second part that basically does the same; however, it starts matching an L.

After executing the search method, it gets regular expression groups that have been returned calling the groups() function, which in this case will return two groups, one for the highest temperature and the second for the lowest.

Other information that we want to provide to our users is information about wind and humidity. The container element that contains this information has a CSS class called today_nowcard-sidecar:

side = container.find('div', class_='today_nowcard-sidecar')
wind, humidity = self._get_additional_info(side)

We just find the container and pass it into the _get_additional_info method that will loop through the children elements of the container, extracting the text and finally returning the results for us.

Finally, the last part of this method:

curr_temp = self._clear_str_number(weatherinfo['today_nowcard-temp'])

self._unit_converter.dest_unit = args.unit

td_forecast = Forecast(self._unit_converter.convert(curr_temp),
                       humidity,
                       wind,
                       high_temp=self._unit_converter.convert(
                           high_temp),
                       low_temp=self._unit_converter.convert(
                           low_temp),
                       description=weatherinfo['today_nowcard- 
                        phrase'])

return [td_forecast]

Since the current temperature contains a special character (degree sign) that we don't want to have at this point, we use the _clr_str_number method to pass the today_nowcard-temp item of the weatherinfo dictionary.

Now that we have all the information we need, we construct the Forecast object and return it. Note that we are returning an array here; this is because all other options that we are going to implement (five-day, ten-day, and weekend forecasts) will return a list, so to make it consistent; also to facilitate when we will have to display this information on the terminal, we are also returning a list.

Another thing to note is that we are making use of the convert method of our UnitConverter to convert all the temperatures to the unit selected in the command line.

When running the command again:

$ python -m weatherterm -u Fahrenheit -a SWXX2372:1:SW -p WeatherComParser -td

You should see an output similar to this:

Congratulations! You have implemented your first web scraping application. Next up, let's add the other forecast options.

The site that we are currently scraping the weather forecast from (weather.com) also provides the weather forecast for five and ten days, so in this section, we are going to implement methods to parse these forecast options as well.

The markup of the pages that present data for five and ten days are very similar; they have the same DOM structure and share the same CSS classes, which makes it easier for us to implement just one method that will work for both options. Let's go ahead and add a new method to the wheater_com_parser.py file with the following contents:

def _parse_list_forecast(self, content, args):
    criteria = {
        'date-time': 'span',
        'day-detail': 'span',
        'description': 'td',
        'temp': 'td',
        'wind': 'td',
        'humidity': 'td',
    }

    bs = BeautifulSoup(content, 'html.parser')

    forecast_data = bs.find('table', class_='twc-table')
    container = forecast_data.tbody

    return self._parse(container, criteria)

As I mentioned before, the DOM for the five- and ten-day weather forecasts is very similar, so we create the _parse_list_forecast method, which can be used for both options. First, we define the criteria:

Now that we have the criteria, we create a BeatufulSoup object, passing the content and the html.parser. All the data that we would like to get is on the table with a CSS class named twc-table. We find the table and define the tbody element as a container. Finally, we run the _parse method, passing the container and the criteria that we defined. The return of this function will look something like this:

[{'date-time': 'Today',
  'day-detail': 'SEP 28',
  'description': 'Partly Cloudy',
  'humidity': '78%',
  'temp': '60°50°',
  'wind': 'ESE 10 mph '},
 {'date-time': 'Fri',
  'day-detail': 'SEP 29',
  'description': 'Partly Cloudy',
  'humidity': '79%',
  'temp': '57°48°',
  'wind': 'ESE 10 mph '},
 {'date-time': 'Sat',
  'day-detail': 'SEP 30',
  'description': 'Partly Cloudy',
  'humidity': '77%',
  'temp': '57°49°',
  'wind': 'SE 10 mph '},
 {'date-time': 'Sun',
  'day-detail': 'OCT 1',
  'description': 'Cloudy',
  'humidity': '74%',
  'temp': '55°51°',
  'wind': 'SE 14 mph '},
 {'date-time': 'Mon',
  'day-detail': 'OCT 2',
  'description': 'Rain',
  'humidity': '87%',
  'temp': '55°48°',
  'wind': 'SSE 18 mph '}]

Another method that we need to create is a method that will prepare the data for us, for example, parsing and converting temperature values and creating a Forecast object. Add a new method called _prepare_data with the following content:

def _prepare_data(self, results, args):
    forecast_result = []

    self._unit_converter.dest_unit = args.unit

    for item in results:
        match = self._temp_regex.search(item['temp'])
        if match is not None:
            high_temp, low_temp = match.groups()

        try:
            dateinfo = item['weather-cell']
            date_time, day_detail = dateinfo[:3], dateinfo[3:]
            item['date-time'] = date_time
            item['day-detail'] = day_detail
        except KeyError:
            pass

        day_forecast = Forecast(
            self._unit_converter.convert(item['temp']),
            item['humidity'],
            item['wind'],
            high_temp=self._unit_converter.convert(high_temp),
            low_temp=self._unit_converter.convert(low_temp),
            description=item['description'].strip(),
            forecast_date=f'{item["date-time"]} {item["day-
             detail"]}',
            forecast_type=self._forecast_type)
        forecast_result.append(day_forecast)

    return forecast_result

This method is quite simple. First, loop through the results and apply the regex that we created to split the high and low temperatures stored in item['temp']. If there's a match, it will get the groups and assign the value to high_temp and low_temp.

After that, we create a Forecast object and append it to a list that will be returned later on.

Lastly, we add the method that will be invoked when the -5d or -10d flag is used. Create another method called _five_and_ten_days_forecast with the following contents:

def _five_and_ten_days_forecast(self, args):
    content = self._request.fetch_data(args.forecast_option.value, args.area_code)
    results = self._parse_list_forecast(content, args)
    return self._prepare_data(results)

This method only fetches the contents of the page passing the forecast_option value and the area code, so it will be possible to build the URL to perform the request. When the data is returned, we pass it down to the _parse_list_forecast, which will return a list of Forecast objects (one for each day); finally, we prepare the data to be returned using the _prepare_data method.

Before we run the command, we need to enable this option in the command line tool that we implemented; go over to the __main__.py file, and, just after the definition of the -td flag, add the following code:

argparser.add_argument('-5d', '--fivedays',
                       dest='forecast_option',
                       action='store_const',
                       const=ForecastType.FIVEDAYS,
                       help='Shows the weather forecast for the next         
                       5 days')

Now, run the application again, but this time using the -5d or --fivedays flag:

$ python -m weatherterm -u Fahrenheit -a SWXX2372:1:SW -p WeatherComParser -5d

It will produce the following output:

>> [Today SEP 28]
    High 60° / Low 50° (Partly Cloudy)
    Wind: ESE 10 mph / Humidity: 78%

>> [Fri SEP 29]
    High 57° / Low 48° (Partly Cloudy)
    Wind: ESE 10 mph / Humidity: 79%

>> [Sat SEP 30]
    High 57° / Low 49° (Partly Cloudy)
    Wind: SE 10 mph / Humidity: 77%

>> [Sun OCT 1]
    High 55° / Low 51° (Cloudy)
    Wind: SE 14 mph / Humidity: 74%

>> [Mon OCT 2]
    High 55° / Low 48° (Rain)
    Wind: SSE 18 mph / Humidity: 87%

To wrap this section up, let's include the option to get the weather forecast for the next ten days as well, in the __main__.py file, just below the -5d flag definition. Add the following code:

argparser.add_argument('-10d', '--tendays',
                       dest='forecast_option',
                       action='store_const',
                       const=ForecastType.TENDAYS,
                       help='Shows the weather forecast for the next  
                       10 days')

If you run the same command as we used to get the five-day forecast but replace the -5d flag with -10d, like so:

$ python -m weatherterm -u Fahrenheit -a SWXX2372:1:SW -p WeatherComParser -10d

You should see the ten-day weather forecast output:

>> [Today SEP 28]
    High 60° / Low 50° (Partly Cloudy)
    Wind: ESE 10 mph / Humidity: 78%

>> [Fri SEP 29]
    High 57° / Low 48° (Partly Cloudy)
    Wind: ESE 10 mph / Humidity: 79%

>> [Sat SEP 30]
    High 57° / Low 49° (Partly Cloudy)
    Wind: SE 10 mph / Humidity: 77%

>> [Sun OCT 1]
    High 55° / Low 51° (Cloudy)
    Wind: SE 14 mph / Humidity: 74%

>> [Mon OCT 2]
    High 55° / Low 48° (Rain)
    Wind: SSE 18 mph / Humidity: 87%

>> [Tue OCT 3]
    High 56° / Low 46° (AM Clouds/PM Sun)
    Wind: S 10 mph / Humidity: 84%

>> [Wed OCT 4]
    High 58° / Low 47° (Partly Cloudy)
    Wind: SE 9 mph / Humidity: 80%

>> [Thu OCT 5]
    High 57° / Low 46° (Showers)
    Wind: SSW 8 mph / Humidity: 81%

>> [Fri OCT 6]
    High 57° / Low 46° (Partly Cloudy)
    Wind: SW 8 mph / Humidity: 76%

>> [Sat OCT 7]
    High 56° / Low 44° (Mostly Sunny)
    Wind: W 7 mph / Humidity: 80%

>> [Sun OCT 8]
    High 56° / Low 44° (Partly Cloudy)
    Wind: NNE 7 mph / Humidity: 78%

>> [Mon OCT 9]
    High 56° / Low 43° (AM Showers)
    Wind: SSW 9 mph / Humidity: 79%

>> [Tue OCT 10]
    High 55° / Low 44° (AM Showers)
    Wind: W 8 mph / Humidity: 79%

>> [Wed OCT 11]
    High 55° / Low 42° (AM Showers)
    Wind: SE 7 mph / Humidity: 79%

>> [Thu OCT 12]
    High 53° / Low 43° (AM Showers)
    Wind: NNW 8 mph / Humidity: 87%

As you can see, the weather was not so great here in Sweden while I was writing this book.

The last weather forecast option that we are going to implement in our application is the option to get the weather forecast for the upcoming weekend. This implementation is a bit different from the others because the data returned by the weekend's weather is slightly different from today's, five, and ten days weather forecast.

The DOM structure is different and some CSS class names are different as well. If you remember the previous methods that we implemented, we always use the _parser method, which gives us arguments such as the container DOM and a dictionary with the search criteria. The return value of that method is also a dictionary where the key is the class name of the DOM that we were searching and the value is the text within that DOM element. 

Since the CSS class names of the weekend page are different, we need to implement some code to get that array of results and rename all the keys so the _prepare_data function can use scraped results properly.

With that said, let's go ahead and create a new file in the weatherterm/core directory called mapper.py with the following contents:

class Mapper:

    def __init__(self):
        self._mapping = {}

    def _add(self, source, dest):
        self._mapping[source] = dest

    def remap_key(self, source, dest):
        self._add(source, dest)

    def remap(self, itemslist):
        return [self._exec(item) for item in itemslist]

    def _exec(self, src_dict):
        dest = dict()

        if not src_dict:
            raise AttributeError('The source dictionary cannot be  
            empty or None')

        for key, value in src_dict.items():
            try:
                new_key = self._mapping[key]
                dest[new_key] = value
            except KeyError:
                dest[key] = value
        return dest

The Mapper class gets a list with dictionaries and renames specific keys that we would like to rename. The important methods here are remap_key and remap. The remap_key gets two arguments, source and dest. source is the key that we wish to rename and dest is the new name for that key. The remap_key method will add it to an internal dictionary called _mapping, which will be used later on to look up the new key name.

The remap method simply gets a list containing the dictionaries and, for every item on that list, it calls the _exec method that first creates a brand new dictionary, then checks whether the dictionary is empty. In that case, it raises an AttributeError.

If the dictionary has keys, we loop through its items, search for whether the current item's key has a new name in the mapping dictionary. If the new key name is found, will to create a new item with the new key name; otherwise, we just keep the old name. After the loop, the list is returned with all the dictionaries containing the keys with a new name.

Now, we just need to add it to the __init__.py file in the weatherterm/core directory:

from .mapper import Mapper

And, in the weather_com_parser.py file in weatherterm/parsers, we need to import the Mapper:

from weatherterm.core import Mapper

With the mapper in place, we can go ahead and create the _weekend_forecast method in the weather_com_parser.py file, like so:

def _weekend_forecast(self, args):
    criteria = {
        'weather-cell': 'header',
        'temp': 'p',
        'weather-phrase': 'h3',
        'wind-conditions': 'p',
        'humidity': 'p',
    }

    mapper = Mapper()
    mapper.remap_key('wind-conditions', 'wind')
    mapper.remap_key('weather-phrase', 'description')

    content = self._request.fetch_data(args.forecast_option.value,
                                       args.area_code)

    bs = BeautifulSoup(content, 'html.parser')

    forecast_data = bs.find('article', class_='ls-mod')
    container = forecast_data.div.div

    partial_results = self._parse(container, criteria)
    results = mapper.remap(partial_results)

    return self._prepare_data(results, args)

The method starts off by defining the criteria in exactly the same way as the other methods; however, the DOM structure is slightly different and some of the CSS names are also different:

As you can see, to make it play nicely with the _prepare_data method, we will need to rename some keys in the dictionaries in the result set—wind-conditions should be wind and weather-phrase should be the description.

Luckily, we have introduced the Mapper class to help us out:

mapper = Mapper()
mapper.remap_key('wind-conditions', 'wind')
mapper.remap_key('weather-phrase', 'description')

We create a Mapper object and say, remap wind-conditions to wind and weather-phrase to description:

content = self._request.fetch_data(args.forecast_option.value,
                                   args.area_code)

bs = BeautifulSoup(content, 'html.parser')

forecast_data = bs.find('article', class_='ls-mod')
container = forecast_data.div.div

partial_results = self._parse(container, criteria)

We fetch all the data, create a BeautifulSoup object using the html.parser, and find the container element that contains the children elements that we are interested in. For the weekend forecast, we are interested in getting the article element with a CSS class called ls-mod and within that article we go down to the first child element, which is a DIV, and gets its first child element, which is also a DIV element.

The HTML should look something like this:

<article class='ls-mod'>
  <div>
    <div>
      <!-- this DIV will be our container element -->
    </div>
  </div>
</article>

That's the reason we first find the article, assign it to forecast_data, and then use forecast_data.div.div so we get the DIV element we want.

After defining the container, we pass it to the _parse method together with the container element; when we get the results back, we simply need to run the remap method of the Mapper instance, which will normalize the data for us before we call _prepare_data.

Now, the last detail before we run the application and get the weather forecast for the weekend is that we need to include the --w and --weekend flag to the ArgumentParser. Open the __main__.py file in the weatherterm directory and, just below the --tenday flag, add the following code:

argparser.add_argument('-w', '--weekend',
                       dest='forecast_option',
                       action='store_const',
                       const=ForecastType.WEEKEND,
                       help=('Shows the weather forecast for the 
                             next or '
                             'current weekend'))

Great! Now, run the application using the -w or --weekend flag:

>> [Fri SEP 29]
    High 13.9° / Low 8.9° (Partly Cloudy)
    Wind: ESE 10 mph / Humidity: 79%

>> [Sat SEP 30]
    High 13.9° / Low 9.4° (Partly Cloudy)
    Wind: SE 10 mph / Humidity: 77%

>> [Sun OCT 1]
    High 12.8° / Low 10.6° (Cloudy)
    Wind: SE 14 mph / Humidity: 74%

Note that this time, I used the -u flag to choose Celsius. All the temperatures in the output are represented in Celsius instead of Fahrenheit.