Generating a warming/cooling filter

When we perceive images, our brain picks up on a number of subtle clues to infer important details about the scene. For example, in broad daylight, highlights may have a slightly yellowish tint because they are in direct sunlight, whereas shadows may appear slightly bluish due to the ambient light of the blue sky. When we view an image with such color properties, we might immediately think of a sunny day.

This effect is no mystery to photographers, who sometimes purposely manipulate the white balance of an image to convey a certain mood. Warm colors are generally perceived as more pleasant, whereas cool colors are associated with night and drabness.

To manipulate the perceived color temperature of an image, we will implement a curve filter. These filters control how color transitions appear between different regions of an image, allowing us to subtly shift the color spectrum without adding an unnatural-looking overall tint to the image.

Color manipulation via curve shifting

A curve filter is essentially a function, y = f(x), that maps an input pixel value x to an output pixel value y. The curve is parameterized by a set of n+1 anchor points, as follows: {(x_0,y_0), (x_1,y_1), ..., (x_n, y_n)}.

Each anchor point is a pair of numbers that represent the input and output pixel values. For example, the pair (30, 90) means that an input pixel value of 30 is increased to an output value of 90. Values between anchor points are interpolated along a smooth curve (hence the name curve filter).

Such a filter can be applied to any image channel, be it a single grayscale channel or the R, G, and B channels of an RGB color image. Thus, for our purposes, all values of x and y must stay between 0 and 255.

For example, if we wanted to make a grayscale image slightly brighter, we could use a curve filter with the following set of control points: {(0,0), (128, 192), (255,255)}. This would mean that all input pixel values except 0 and 255 would be increased slightly, resulting in an overall brightening effect of the image.

If we want such filters to produce natural-looking images, it is important to respect the following two rules:

• Every set of anchor points should include (0,0) and (255,255). This is important in order to prevent the image from appearing as if it has an overall tint, as black remains black and white remains white.

• The function f(x) should be monotonously increasing. In other words, with increasing x, f(x) either stays the same or increases (that is, it never decreases). This is important for making sure that shadows remain shadows and highlights remain highlights.

Implementing a curve filter by using lookup tables

Curve filters are computationally expensive, because the values of f(x) must be interpolated whenever x does not coincide with one of the prespecified anchor points. Performing this computation for every pixel of every image frame that we encounter would have dramatic effects on performance.

Instead, we make use of a lookup table. Since there are only 256 possible pixel values for our purposes, we need to calculate f(x) only for all the 256 possible values of x. Interpolation is handled by the UnivariateSpline function of the scipy.interpolate module, as shown in the following code snippet:

from scipy.interpolate import UnivariateSpline

def _create_LUT_8UC1(self, x, y):
  spl = UnivariateSpline(x, y)
  return spl(xrange(256))

The return argument of the function is a 256-element list that contains the interpolated f(x) values for every possible value of x.

All we need to do now is come up with a set of anchor points, (x_i, y_i), and we are ready to apply the filter to a grayscale input image (img_gray):

import cv2
import numpy as np

x = [0, 128, 255]
y = [0, 192, 255]
myLUT = _create_LUT_8UC1(x, y)
img_curved = cv2.LUT(img_gray, myLUT).astype(np.uint8)

The result looks like this (the original image is on the left, and the transformed image is on the right):

Designing the warming/cooling effect

With the mechanism to quickly apply a generic curve filter to any image channel in place, we now turn to the question of how to manipulate the perceived color temperature of an image. Again, the final code will have its own class in the filters module.

If you have a minute to spare, I advise you to play around with the different curve settings for a while. You can choose any number of anchor points and apply the curve filter to any image channel you can think of (red, green, blue, hue, saturation, brightness, lightness, and so on). You could even combine multiple channels, or decrease one and shift another to a desired region. What will the result look like?

However, if the number of possibilities dazzles you, take a more conservative approach. First, by making use of our _create_LUT_8UC1 function developed in the preceding steps, let's define two generic curve filters, one that (by trend) increases all pixel values of a channel, and one that generally decreases them:

class WarmingFilter:

  def __init__(self):
    self.incr_ch_lut = _create_LUT_8UC1([0, 64, 128, 192, 256],
      [0, 70, 140, 210, 256])
    self.decr_ch_lut = _create_LUT_8UC1([0, 64, 128, 192, 256],
      [0, 30,  80, 120, 192])

The easiest way to make an image appear as if it was taken on a hot, sunny day (maybe close to sunset), is to increase the reds in the image and make the colors appear vivid by increasing the color saturation. We will achieve this in two steps:

1. Increase the pixel values in the R channel and decrease the pixel values in the B channel of an RGB color image using incr_ch_lut and decr_ch_lut, respectively:

   def render(self, img_rgb):
       c_r, c_g, c_b = cv2.split(img_rgb)
       c_r = cv2.LUT(c_r, self.incr_ch_lut).astype(np.uint8)
       c_b = cv2.LUT(c_b, self.decr_ch_lut).astype(np.uint8)
       img_rgb = cv2.merge((c_r, c_g, c_b))

2. Transform the image into the HSV color space (H means hue, S means saturation, and V means value), and increase the S channel using incr_ch_lut. This can be achieved with the following function, which expects an RGB color image as input:

   c_b = cv2.LUT(c_b, decrChLUT).astype(np.uint8)
   
   # increase color saturation
   c_h, c_s, c_v = cv2.split(cv2.cvtColor(img_rgb, 
       cv2.COLOR_RGB2HSV))
   c_s = cv2.LUT(c_s, self.incr_ch_lut).astype(np.uint8)
   return cv2.cvtColor(cv2.merge((c_h, c_s, c_v)), 
       cv2.COLOR_HSV2RGB)

The result looks like what is shown here:

Analogously, we can define a cooling filter that increases the pixel values in the B channel, decreases the pixel values in the R channel of an RGB image, converts the image into the HSV color space, and decreases color saturation via the S channel:

class CoolingFilter:

    def render(self, img_rgb):

        c_r, c_g, c_b = cv2.split(img_rgb)
        c_r = cv2.LUT(c_r, self.decr_ch_lut).astype(np.uint8)
        c_b = cv2.LUT(c_b, self.incr_ch_lut).astype(np.uint8)
        img_rgb = cv2.merge((c_r, c_g, c_b))

        # decrease color saturation
        c_h, c_s, c_v = cv2.split(cv2.cvtColor(img_rgb, cv2.COLOR_RGB2HSV))
        c_s = cv2.LUT(c_s, self.decr_ch_lut).astype(np.uint8)
        return cv2.cvtColor(cv2.merge((c_h, c_s, c_v)), cv2.COLOR_HSV2RGB)

Now, the result looks like this: