Making subtle color shifts with curves

In this article, written by Joseph Howse, author of the book Android Application Programming with OpenCV3, we understand that, when looking at a scene, we may pick up subtle cues from the way colors shift between different image regions. For example, on a clear day outside, shadows have a slightly blue tint due to the ambient light reflected from the blue sky, while highlights have a slightly yellow tint because they are in direct sunlight. When we see bluish shadows and yellowish highlights in a photograph, we may get a "warm and sunny" feeling. This effect may be natural, or it may be exaggerated by a filter.

(For more resources related to this topic, see here.)

Curve filters are useful for this type of manipulation. A curve filter is parameterized by sets of control points. For example, there might be one set of control points for each color channel. Each control point is a pair of numbers that represents the input and output values of the given channel. For example, the pair (128, 180) means that a value of 128 in the given color channel is brightened to become a value of 180. Values between the control points are interpolated along a curve (hence, the name, curve filter). In GIMP, a curve with the control points (0, 0), (128, 180), and (255, 255) is visualized, as shown in the following screenshot:

Android Application Programming with OpenCV 3

The x axis shows the input values ranging from 0 to 255, while the y axis shows the output values over the same range. Besides showing the curve, the graph shows the line y = x (no change) for comparison.

Curvilinear interpolation helps to ensure that color transitions are smooth, not abrupt. Thus, a curve filter makes it relatively easy to create subtle, natural-looking effects. We may define an RGB curve filter as follows, in pseudocode:

dst.r = funcR(src.r) where funcR interpolates pointsR
dst.g = funcG(src.g) where funcG interpolates pointsG
dst.b = funcB(src.b) where funcB interpolates pointsB

For now, we will work with RGB and RGBA curve filters and with channel values that range from 0 to 255. If we want such a curve filter to produce natural-looking results, we should use the following rules of thumb:

  • Every set of control points should include (0, 0) and (255, 255). This way, black remains black, white remains white, and the image does not appear to have an overall tint.
  • As the input value increases, the output value should always increase too. (Their relationship should be monotonically increasing.) This way, shadows remain shadows, highlights remain highlights, and the image does not appear to have inconsistent lighting or contrast.

OpenCV does not provide curvilinear interpolation functions, but the Apache Commons Math library does. This library provides interfaces called UnivariateInterpolator and UnivariateFunction, which have implementations including LinearInterpolator, SplineInterpolator, LinearFunction, and PolynomialSplineFunction. (Splines are a type of curve.) UnivariateInterpolator has an instance method, interpolate(double[] xval, double[] yval), which takes arrays of input and output values for the control points and returns a UnivariateFunction object. The UnivariateFunction object can provide interpolated values via the method value(double x).

API documentation for Apache Commons Math is available at http://commons.apache.org/proper/commons-math/apidocs/.

These interpolation functions are computationally expensive. We do not want to run them again and again for every channel of every pixel in every frame. Fortunately, we do not have to. There are only 256 possible input values per channel, so it is practical to precompute all possible output values and store them in a lookup table. For OpenCV's purposes, a lookup table is a Mat object whose indices represent input values and whose elements represent output values. The lookup can be performed using the static method Core.LUT(Mat src, Mat lut, Mat dst). In pseudocode, dst = lut[src]. The number of elements in lut should match the range of values in src, and the number of channels in lut should match the number of channels in src.

Now, using Apache Commons Math and OpenCV, let's implement a curve filter for RGBA images with channel values ranging from 0 to 255. Open CurveFilter.java and write the following code:

public class CurveFilter implements Filter {
// The lookup table.
private final Mat mLUT = new MatOfInt();
public CurveFilter(
   final double[] vValIn, final double[] vValOut,
   final double[] rValIn, final double[] rValOut,
   final double[] gValIn, final double[] gValOut,
   final double[] bValIn, final double[] bValOut) {
   // Create the interpolation functions.
   UnivariateFunction vFunc = newFunc(vValIn, vValOut);
   UnivariateFunction rFunc = newFunc(rValIn, rValOut);
   UnivariateFunction gFunc = newFunc(gValIn, gValOut);
   UnivariateFunction bFunc = newFunc(bValIn, bValOut);
   // Create and populate the lookup table.
   mLUT.create(256, 1, CvType.CV_8UC4);
   for (int i = 0; i < 256; i++) {
     final double v = vFunc.value(i);
     final double r = rFunc.value(v);
     final double g = gFunc.value(v);
     final double b = bFunc.value(v);
     mLUT.put(i, 0, r, g, b, i); // alpha is unchanged
   }
}
@Override
public void apply(final Mat src, final Mat dst) {
   // Apply the lookup table.
   Core.LUT(src, mLUT, dst);
}
private UnivariateFunction newFunc(final double[] valIn,
   final double[] valOut) {
   UnivariateInterpolator interpolator;
   if (valIn.length > 2) {
     interpolator = new SplineInterpolator();
   } else {
     interpolator = new LinearInterpolator();
   }
   return interpolator.interpolate(valIn, valOut);
}
}

CurveFilter stores the lookup table in a member variable. The constructor method populates the lookup table based on the four sets of control points that are taken as arguments. In addition to a set of control points for each of the RGB channels, the constructor also takes a set of control points for the image's overall brightness, just for convenience. A helper method, newFunc, creates an appropriate interpolation function (linear or spline) for each set of control points. Then, we iterate over the possible input values and populate the lookup table.

The apply method is a one-liner. It simply uses the precomputed lookup table with the given source and destination matrices.

CurveFilter can be extended in a subclass to define a filter with a specific set of control points. For example, let's open PortraCurveFilter.java and write the following code:

public class PortraCurveFilter extends CurveFilter {
public PortraCurveFilter() {
   super(
     new double[] { 0, 23, 157, 255 }, // vValIn
     new double[] { 0, 20, 173, 255 }, // vValOut
     new double[] { 0, 69, 213, 255 }, // rValIn
     new double[] { 0, 69, 218, 255 }, // rValOut
     new double[] { 0, 52, 189, 255 }, // gValIn
     new double[] { 0, 47, 196, 255 }, // gValOut
     new double[] { 0, 41, 231, 255 }, // bValIn
     new double[] { 0, 46, 228, 255 }); // bValOut
}
}

This filter brightens the image, makes shadows cooler (more blue), and makes highlights warmer (more yellow). It produces flattering skin tones and tends to make things look sunnier and cleaner. It resembles the color characteristics of a brand of photo film called Kodak Portra, which was often used for portraits.

The code for our other three channel mixing filters is similar. The ProviaCurveFilter class uses the following arguments for its control points:

     new double[] { 0, 255 }, // vValIn
     new double[] { 0, 255 }, // vValOut
     new double[] { 0, 59, 202, 255 }, // rValIn
     new double[] { 0, 54, 210, 255 }, // rValOut
     new double[] { 0, 27, 196, 255 }, // gValIn
     new double[] { 0, 21, 207, 255 }, // gValOut
     new double[] { 0, 35, 205, 255 }, // bValIn
     new double[] { 0, 25, 227, 255 }); // bValOut

The effect of this filter is to increase the contrast between shadows and highlights and make the image slightly cooler (bluish) throughout most of the tones. Sky, water, and shade are accentuated more than the sun. It resembles a brand of photo film called Fuji Provia, which was often used for landscapes. For example, the following photo has been taken on Provia film, which accentuates the bluish, misty background in an otherwise sunny scene:

Android Application Programming with OpenCV 3

The VelviaCurveFilter class uses the following arguments for its control points:

     new double[] { 0, 128, 221, 255 }, // vValIn
     new double[] { 0, 118, 215, 255 }, // vValOut
     new double[] { 0, 25, 122, 165, 255 }, // rValIn
     new double[] { 0, 21, 153, 206, 255 }, // rValOut
     new double[] { 0, 25, 95, 181, 255 }, // gValIn
     new double[] { 0, 21, 102, 208, 255 }, // gValOut
     new double[] { 0, 35, 205, 255 }, // bValIn
     new double[] { 0, 25, 227, 255 }); // bValOut

The effect of this filter is to produce deep shadows and vivid colors. It resembles a brand of photo film called Fuji Velvia, which was often used to depict landscapes, with azure skies in daytime or crimson clouds at sunset. The next photo has been taken on Velvia film and, in this sunny scene, we can see Velvia's distinctive deep shadows and azure skies (or slate gray skies in the black-and-white print edition):

Android Application Programming with OpenCV 3

Finally, the CrossProcessCurveFilter class uses the following arguments for its control points:

     new double[] { 0, 255 }, // vValIn
     new double[] { 0, 255 }, // vValOut
     new double[] { 0, 56, 211, 255 }, // rValIn
     new double[] { 0, 22, 255, 255 }, // rValOut
     new double[] { 0, 56, 208, 255 }, // gValIn
     new double[] { 0, 39, 226, 255 }, // gValOut
     new double[] { 0, 255 }, // bValIn
     new double[] { 20, 235 }); // bValOut

The effect is a strong, blue or greenish-blue tint in shadows and a strong, yellow or greenish-yellow tint in highlights. It resembles a film processing technique called cross-processing, which was sometimes used to produce grungy-looking photos of fashion models, pop stars, and so on.

For a good discussion of how to emulate various brands of photo film, see Petteri Sulonen's blog at http://www.prime-junta.net/pont/How_to/100_Curves_and_Films/_Curves_and_films.html. The control points that we use are based on the examples given in this article.

The following strip of screenshots is a showcase of our curve filters. Some of the differences are subtle. From left to right, we see an unfiltered image, followed by images filtered with PortraCurveFilter, ProviaCurveFilter, VelviaCurveFilter, and CrossProcessCurveFilter:

Android Application Programming with OpenCV 3

The differences between these curve filters are not obvious in black-and-white prints.

Curve filters are a convenient tool for manipulating color and contrast, but they are limited insofar as each destination pixel is affected by only a single input pixel. Next, we will examine a more flexible family of filters, which enable each destination pixel to be affected by a neighborhood of input pixels.

Summary

Second Sight now has some functionality that is more interesting than just reading and sharing camera data. Several filters can be selected and combined to give a stylized or vintage look to our photos. These filters are efficient enough to apply to live video too, so we use them in preview mode and on saved photos.

Although photo filters are fun, they are only the most basic use of OpenCV. Before we can truly say we have made a computer vision application, we need to make the app respond differently depending on what it is seeing.

Resources for Article:


Further resources on this subject:


You've been reading an excerpt of:

Android Application Programming with OpenCV 3

Explore Title
comments powered by Disqus