Learning OpenCV 3 Application Development:

There are countless sophisticated instruments that aid us in the process of acquiring images from nature. At the most basic level, they work by capturing light rays as they enter through the aperture of the instrument's lens and fall on a photographic plate. Depending on the orientation, illumination, and other parameters of the photo-capturing device, the amount of light that falls on each spatial coordinate of the film differs. This variation in the intensity of light falling on the film is encoded as pixel values when the image is stored in a digital format. Therefore, the information stored by a pixel is nothing more than a quantitative measure of the intensity of light that illuminated that particular spatial coordinate while the image was being captured. What this essentially means is that any image that you see, when represented digitally, is reduced to a two-dimensional grid of values where each pixel in the image is assigned a numerical value that is directly proportional to the intensity of light falling on that pixel in the natural image.

Now we come to the issue of encoding light intensity in pixel values. If you have studied a programming language before, you might be aware that the range and the type of values that you can store in any data structure are closely linked to the data type. A single bit can represent two values: 0 and 1. Eight bits (also known as a byte) can accommodate different values. Going further along, an int (represented using 32 bits in most architectures) data type has the capacity to represent roughly 4.29 billion different entries. Extending the same logic to digital images, the range of values that can be used to represent the pixel intensities depends on the data type we select for storing the image. In the world of image processing, the term color space or color depth is used in place of data type.

The most common and simplest color space for representing images is using 8 bits to represent the value of each pixel. This means that each pixel can have any value between 0 and 255 (inclusive). Images made up of such color spaces are called grayscale images. By convention, 0 represents black, 255 represents white, and each of the other values between 0 and 255 stand for a different shade of gray. The following figure demonstrates such an 8-bit color space. As we move from left to right in the following figure, the grayscale values in the image gradually change from 0 to 255:

So, if we have a grayscale image, such as the following one, then to a digital medium, it is merely a matrix of values-where each element of the matrix is a grayscale value between 0 (black) to 255 (white). This grid of pixel intensity values is represented for a tiny sub-section of the image (a portion of one of the wing mirrors of the car).

We have seen that using 8 bits is sufficient to represent grayscale images in digital media. But how do we represent colors? This brings us to the concept of color channels. A majority of the images that you come across are colored as opposed to grayscale. In the case of the image we just saw, each pixel is associated with a single intensity value (between 0 and 255). For color images, each pixel has three values or components: the red (R), green (G), and blue (B) components. It is a well-known fact that all possible colors can be represented as a combination of the R, G, and B components, and hence, the triplet of intensity values at each pixel are sufficient to represent the entire spectrum of colors in the image. Also, note that each of the three R, G, and B values at every pixel are stored using 8 bits, which makes it 8 x 3 = 24 bits per pixel. This means that the color space now increases to more than 16 million colors from a mere 256. This is the reason color images store much more information than their grayscale counterparts.

Conceptually, the color image is not treated as having a triplet of intensity values at each pixel. Rather, a more convenient form of representation is adopted. The image is said to possess three independent color channels: the R, G, and B channels. Now, since we are using 8 bits per pixel per channel, each of the three channels are grayscale images in themselves!

The first two arguments define the dimensions of the data matrix, that is, rows and columns, respectively. So the previous example will create a Mat object with a data matrix comprising 20 rows and 15 columns, which means a total of 20 x 15 = 300 elements. Often, you will see Mat declarations where both of these values are combined into a single argument: the Size object. The Size object, more specifically, the Size_ template class, is an OpenCV specific class that allows us to specify sizes for images and rectangles. It has two members: width and height. So, if you are using a Size object to specify the dimensions of a Mat, the height and width correspond to the number of rows and columns, respectively. The same Mat instantiation using a Size object is given as follows:

Mat M(Size(15, 20), CV_8UC3, Scalar(0,0,255)); 

There are a couple of things that are noteworthy regarding the preceding line of code. First, note that the number of rows and columns are in the reverse order with respect to the previous instantiation. This is because the constructor for the Size_ class accepts the arguments in this order-width and height. Second, note that although the class is templatized and named Size_, in the declaration, we simply use Size. This is due to the fact that OpenCV has defined some aliases as follows:

typedef Size_<int> Size2i; 
typedef Size2i Size; 

This basically means that writing Size is equivalent to saying Size2i, which in turn is the same as Size_<int>.

The next argument to the Mat declaration statement discussed earlier is for the type. This parameter defines the type of values that the data matrix of the Mat object would store. The choice of this parameter becomes important because it controls the amount of space needed to store the Mat object in memory. OpenCV has its own types defined. A mapping between the OpenCV types and C++ data types is given in the following table:

16, 32, and 64 represent the number of bits used for storing a value of that data type. U, S, and F stand for unsigned, short, and float, respectively. Using these two pieces of information, we can easily deduce the range of values for each data type, as given in the right-most column of the table.

You will notice a C followed by a number in the types used to declare our Mat objects (for example, CV_8UC3). The C here stands for channel, and the integer following it gives you the number of channels in the image. Given a multi-channel, RGB image, OpenCV provides you with a split() function that separates the three channels. Here is a short code snippet that demonstrates this:

Mat color_image = imread("lena.jpg", IMREAD_COLOR); 
vector<Mat> channels; 
split(color_image, channels); 
 
imshow("Blue", channels[0]); 
imshow("Green", channels[1]); 
imshow("Red", channels[2]); 
waitKey(0);

By looking at the complete OpenCV type (along with the number of channels) and the Mat object dimensions, we can actually calculate the number of bits that would be required to store all the pixel values in memory. For example, let's say we have a 100 x 100 Mat object of type CV_8UC3. Each pixel value will take 8 bits and there will be three such values for each pixel (three channels). That takes it to 24 bits per pixel. There are 100 x 100 = 10,000 pixels in total, which means a total space of (24 x 10,000) bits = 30 kilobytes. Keep in mind that this is the space used up by the grid of pixel values and does not include the header. The overall size of the Mat object will be higher, but not by a significant amount (the size of the data matrix is substantially larger than the size of a Mat header).

By looking at the range of data types available for declaring Mat objects, it's natural to think about the utility of all the different types. For storing and representing images, only CV_8UC1 and CV_8UC3 make sense, the former for grayscale images and the latter for images in the RGB color space. As stated earlier, in OpenCV, the Mat object is used for much more than an image store. For applications where Mat is best treated as a multidimensional numerical array, the other types make sense. However, irrespective of whether the Mat object serves as an image store or as a data structure, its importance and ubiquity inside the world of OpenCV is undeniable.

The last argument is the default value for the data matrix of the Mat object. You will have noticed the use of yet another OpenCV specific data structure: Scalar. The Scalar_ class allows you to store a vector of at most four values. You might be wondering about the utility of restricting the size of a vector to just four. There are several use cases within OpenCV where we might require working with one, two, or three values (and not more than that). For example, we have just learnt that each pixel in an RGB image is represented using three values, one each for the R, G, and B channels. In such a scenario, the Scalar object provides a convenient method to pass the group of three values to the Mat object constructor, as has been done in the example under consideration. One important thing to note is that OpenCV reads the color channels in the reverse order-B, G, and R. This means that passing Scalar(255, 0, 0) would refer to blue, whereas Scalar(0, 0, 255) is red. Any combination of the three would then represent one of the 16 million+ colors. If, at this point of time, you are wondering about providing default values for a grayscale image, OpenCV allows what is intuitive. A simple Scalar(0) or Scalar(255) will color all pixels black or white, respectively, by default in a grayscale image. What I mean to say is that the constructor for the Scalar object is flexible enough to accept one, two, three, or even four values. If you are wondering about the discrepancy in the class names, Scalar_ and Scalar, similar to the Size_ class, OpenCV defines the following alias to make our code less verbose:

typedef Scalar_<double> Scalar; 

The initialization method that we discussed here involved passing all the three pieces of information as arguments:

However, the Mat class allows greater flexibility in declaring objects. You do not have to specify all the three mentioned earlier. The Mat class has some overloaded constructors that allow you to declare objects even if you simply specify the following:

Here are some of the constructor declarations from the implementation of the Mat class:

Mat () 
Mat (int rows, int cols, int type) 
Mat (Size size, int type) 
Mat (int rows, int cols, int type, const Scalar &s) 
Mat (Size size, int type, const Scalar &s) 

Going by the preceding definitions, we present some sample valid Mat object declarations. You will get a chance to see them being used in the programs that we write as part of this book:

Mat I; 
Mat I(100, 80, CV_8UC1); 
Mat I(Size(80, 100), CV_8UC1); 

Before we finish this section on declaring Mat objects, we will discuss one final technique, that is, creating Mat objects as a region of interest (ROI) from inside an existing Mat object. Often, situations arise where we are interested in a subset of the data from the data matrix of an existing Mat object. Putting it another way, we would like to initialize a new Mat object whose data matrix is a submatrix of the existing Mat object. The constructor for such an initialization is given as Mat (const Mat &m, const Rect &roi). A sample statement that invokes such a constructor is given as follows:

Mat roi_image(original_image, Rect(10, 10, 100, 100)); 

This will create a new Mat object named roi_image by taking the data from the matrix belonging to the existing Mat object, original_image. The submatrix will start from the pixel with coordinates (10, 10) as the upper-left corner and will have dimensions of 100 x 100. All the information pertaining to the size of the ROI has been passed via the Rect object, which is yet another OpenCV specific data structure.

Before we start with the code for traversing Mat objects, we need to understand how (more precisely, in what order) the data matrix stores the pixel values in memory. To do that, we need to introduce the concept of continuity. A data matrix is said to be continuous if all its rows are stored at adjacent memory locations without any gap between the contents of two successive rows. If a matrix is not continuous, it is said to be non-continuous. Now, why do we care if our Mat object's underlying data matrix is continuous or not? Well, as it turns out that iterating a continuous data matrix is much faster than going over a non-continuous one because the former requires a smaller number of memory accesses. Having learnt about the benefits offered by the continuity property of a Mat object's data matrix, how do we take advantage of this feature in our applications? The answer to that can be found in the following code snippet:

int channels = image.channels(); 
int num_rows = image.rows; 
int num_cols = (image.cols * channels); 
 
if (image.isContinuous()) { 
    num_cols = num_cols * num_rows; 
    num_rows = 1; 
} 

This piece of code achieves what I like to call flattening of the data matrix, and this is typically performed as a precursor to the actual image traversal. If the rows of the data matrix are indeed saved in contiguous memory locations, this means that we can treat the entire matrix as a single one-dimensional array. This array will have one row and the number of columns will be equal to (numRows*numCols*numChannels), which is the total number of pixels in the image. The code snippet assumes that the image is an 8-bit Mat object. Also note that the flattening is performed only if the image is continuous.

In the case of non-continuous images, the value of numRows and numCols remain as they are read from the Mat object.

Matrices created by imread(), clone(), or a constructor will always be continuous. In fact, the only time a matrix will not be continuous is when it borrows data from an existing matrix. By borrowing data, I mean when a new matrix is created out of an ROI of a bigger matrix, for example:

Mat big (200, 300, CV_8UC1); 
Mat roi (big, Rect(10, 10, 100, 100)); 
Mat col = big.col(0); 

Both matrices, roi and col, will be non-continuous as they borrow data from big.

Now, we are ready for the actual traversal. As stated earlier, we will discuss a couple of different ways to go about this. The first technique uses the ptr() method of the Mat class. According to the documentation of the Mat::ptr() method, it returns a pointer to the specified matrix row. We specify the row by its 0 based index passed to the function as an argument. So, let's check out the Mat::ptr() method in action:

for (int i = 0; i < numRows; ++i) { 
    uchar* row_ptr = image.ptr<uchar>(i); 
    for (int j = 0; j < numCols; ++j) { 
        // row_ptr[j] will give you access to the pixel value 
        // any sort of computation/transformation is to be performed here 
    } 
} 

What this technique essentially does is acquire the pointer to the start of each row with the statement image.ptr<uchar>(i) and save it in a pointer variable named row_ptr (the outer for loop); loop variable i is used to index the rows of the matrix. Once we have acquired the pointer to an image row, we iterate through the row to access the value of each and every pixel. This is precisely what the inner for loop, which has the j loop variable, accomplishes. What is elegant about this code is that it works in both cases, whether our data matrix is continuous (and flattened) or not. Just think about it; if our matrix were continuous and had been flattened using the code that we discussed a while back, then it would have had a single row (numRows=1) and the number of columns would have been the same as the number of pixels in the image, . This would mean that the outer loop runs only once and we call the Mat::ptr() method once to fetch all the pixels of the image in a single call. And if our matrix hasn't been flattened, then image.ptr<uchar>(i) will be called for each row that makes it a total of numRowstimes. This is also the reason that flattening a matrix is more efficient in terms of time taken.

Let's put together the code for the flattening and traversal of the image to get a complete picture of using the pointer method for Mat object traversal:

void scanImage(Mat& image) { 
    int channels = image.channels(); 
    int num_rows = image.rows; 
    int num_cols = (image.cols * channels); 
 
    if (image.isContinuous()) { 
        num_cols *= num_rows; 
        num_rows = 1; 
} 
 
for (int i = 0; i < num_rows; ++i) { 
    uchar* row_ptr = image.ptr<uchar>(i); 
    for (int j = 0; j < num_cols; ++j) { 
        // Perform operations on pixel value row_ptr[j] 
    } 
} 
} 

So, in summary, the Mat::ptr() method essentially works by fetching the data one row at a time. In that sense, the access method here is sequential: when the data of one of the rows is fetched, we can go over the contents of only that particular row. Accessing a new row necessitates a new fetch call. Flattening the data matrix is just a way to speed up computation, which works by bringing in all the data in a single fetch. This might not be the most aesthetic way of doing things. Your code may sometimes be difficult to understand and/or debug, especially when it comes to handling multi-channel images (you need to know exactly how many columns to skip per pixel while traversing a row). Now, this is where our second approach comes in.

This method relies on the Mat::at() method. As per the OpenCV documentation, the at() method returns a reference to any specified array element. The pixel whose value we are interested in is specified via the row and column index. This approach provides us with a random access to the data matrix. Let's look at an example code in action that uses the at() method to access pixel values. In the following code snippet, assume that I is a single-channel, grayscale image:

for( int i = 0; i < I.rows; ++i) { 
    for( int j = 0; j < I.cols; ++j) { 
        // Matrix elements can be accessed via : I.at<uchar>(i,j) 
    } 
} 

The code looks much simpler, more compact, and easier to read than the earlier approach. We have a couple of for loops: the outer loop (with index variable i) which iterates over the rows and the inner loop (with index variable j) that goes over the columns. As we move over each pixel, we can access its value by calling I.at<uchar>(i,j).

But what about the case when our image is multi-channeled? Let's say that we have a three-channel RGB image that we need to traverse. The code would have a very similar structure but with minor differences. Since our image is now three-channeled, the  uchar data type will not be appropriate for the pixel values. The solution is presented in the following code snippet:

for( int i = 0; i < I.rows; ++i) { 
    for( int j = 0; j < I.cols; ++j) { 
        /**  
        * The B, G and R components for the (i, j)-th pixel can be accessed by: 
        * I.at<Vec3b>(i, j)[0] 
        * I.at<Vec3b>(i, j)[1] 
        * I.at<Vec3b>(i, j)[2] 
        **/ 
    } 
} 

The first thing you notice about the code is the use of what seems like a new OpenCV type named Vec_3b. All you need to know about Vec_3b at this point is that it stands for a vector of three byte values, that is, a vector of three numbers between 0 and 255 (inclusive). And that seems to be the perfect data type for representing what a pixel stands for in a three-channel RGB image (OpenCV always has the right tools made available to its users!). Now that we have established that the type of each value in the data matrix is Vec_3b, which means that the at() method returns a reference to Vec_3b, we can access the individual elements within Vec_3b using the [] operator, just like a C++ array or vector. Now, recall that when we discussed about image channels, we said that the OpenCV stores the R, G, and B components in the reverse order. This would mean that the zeroth, first and second elements of Vec_3b would each refer to the blue, green, and red components of the pixel, respectively. You should be extra careful about this fact as it can be a potential source of errors in your code.

Now, the library has gone a step further to provide another level of convenience for its users. Using the previously mentioned approach, we have to write the name of the data type Vec_3b every time we want to access the value for a particular channel of a particular pixel. In order to avoid that, OpenCV provides us with a template class named Mat_. As always, we demonstrate its use via an example code snippet:

Mat_<Vec3b> _I = I; 
for( int i = 0; i < I.rows; ++i) { 
    for( int j = 0; j < I.cols; ++j ) { 
        /**  
        * The B, G and R components for the (i, j)-th pixel can be accessed by: 
        * _I(i, j)[0] 
        * _I(i, j)[1] 
        * _I(i, j)[2] 
        **/ 
    } 
} 

The first thing we do is declare an object of the Mat_ class and initialize it with our original Mat object. Mat_ is a thin template wrapper over the Mat class. It doesn't contain any extra data fields in addition to what is available with the Mat object. In fact, references to the two classes (Mat and Mat_) can be converted to each other. The only advantage Mat_ offers is the notational convenience of having to skip writing the data type every time we have to access a pixel (this is because the data type has been specified during declaration of the Mat_ object itself).

As stated earlier, the Mat::at() method is suited for random access (it requires both the row and column index), the code is much more readable and clean, but it is slower than the pointer-based approach because the at() method does some range checks each time it is called.

We combine both the code snippets for single as well as multi-channel traversal using Mat::at() and encapsulate that within a single C++ function:

void scanImage(Mat& image) { 
  int channels = image.channels(); 
   
  if (channels == 1) { 
    for( int i = 0; i < I.rows; ++i) { 
      for( int j = 0; j < I.cols; ++j) { 
        // Matrix elements can be accessed via : I.at<uchar>(i,j) 
      } 
    } 
  }  
  else if (channels == 3)  { 
    for( int i = 0; i < I.rows; ++i) { 
      for( int j = 0; j < I.cols; ++j) {  
        // The B, G and R components for the (i, j)-th pixel can be  
        // accessed by: 
        //   I.at<Vec3b>(i, j)[0] 
        //   I.at<Vec3b>(i, j)[1] 
        //   I.at<Vec3b>(i, j)[2] 
      } 
    } 
  } 
} 

This concludes our section on image traversals. But before we move on to the next topic, a few final words on Mat object traversals. We have gone over a lot of different methods to achieve what seems like a very basic task. We have seen the sequential-pointer approach and the random-access technique using the Mat::at() method. Personally, I tend to lean towards the latter due to its aesthetic appeal and a clear distinction between single and multichannel images that leaves no room for confusion. It's usually also safer, due to the range checks that we've mentioned before, and it's also easier to access the surrounding pixels if you need them for processing (something that we would be doing quite a lot from Chapter 2, Image Filtering.

Most of the example programs in the remainder of this book will stick to this too. However, you are encouraged to try out the former approach too, if and when you feel like, and compare the results with the ones shown in the text.

Now, we have been traversing Mat objects and images for quite some time now but haven't really been doing any sort of tangible processing with them. You will have noticed that when it came to the section of code where we had the chance to actually access and/or modify the pixel values, we stopped and hid behind those boring comment blocks that did nothing but tell us more theory about how to code. Very soon, in the next few sections, we are going to remove those comments and fill up that space with some actual code that performs some simple, yet cool transformations on our images!

The identity transformation maps each input pixel to itself in the output. In other words:

T(r)=r

Obviously, this does nothing exciting. In fact, this transformation doesn't do anything at all! The output image is the same as the input image, because every pixel gets mapped to itself in the transformation. Nevertheless, we discuss it here for the sake of completeness.

Implementing a lookup table for identity transformations shouldn't be a hassle at all:

vector getIdentityLUT() { 
  vector<uchar> LUT(256, 0); 
  for (int i = 0; i < 256; ++i) 
    LUT[i] = (uchar)i; 
  return LUT; 
} 

The first line of the function declares and initializes the C++ vector that is going to serve as our lookup table. This same vector is then computed and returned by the function. We had discussed earlier (while talking about lookup tables) that the size of LUT will, in practically all cases, be 256 for the 256 different intensity values in a grayscale image. The for loop traverses over LUT and encodes the transformation. Notice that LUT[i]=i will map every input pixel to itself, thereby implementing the identity transformation.

As stated earlier, one of the secondary benefits of using a lookup table is that it modularizes your code and makes it cleaner. The preceding snippet that we showed for identity transformations only computes and returns the lookup table. You can use a matrix traversal method after this to actually apply the transformation to all the pixels of an image. In fact, we demonstrated this framework in our section on Lookup Tables.

void processImage(Mat& I) { 
  vector<uchar> LUT = getLUT(); 
  for (int i = 0; i < I.rows; ++i) { 
    for (int j = 0; j < I.cols; ++j) 
      I.at<uchar>(i, j) = LUT[I.at<uchar>(i, j)]; 
  } 
} 
 
void main() { 
  Mat image = imread("lena.jpg", IMREAD_GRAYSCALE); 
  Mat processed_image = image.clone(); 
  processImage(processed_image); 
 
  imshow("Input image", image); 
  imshow("Processed Image", processed_image); 
  waitKey(0); 
   
return 0; 
} 

Note that while invoking the processImage() method to do our bidding, we have passed it a clone of the matrix that we just read from the input. This is just so that we are able to compare the changes that the processing has made on our input (not that there is any, in this particular case!). From the next transformation onwards, we are not going to write down the full detailed code (along with the processImage() and main()). We'll focus on the computation of the lookup table because that is what varies from one transformation to the next.

The negative transformation subtracts 255 from the input pixel intensity value and produces that as an output. Mathematically speaking, the negative transformation can be expressed as follows:

s=T(r)=(255-r)

This means that a value of 0 in the input (black) gets mapped to 255 (white) and vice versa. Similarly, lighter shades of gray will yield the corresponding darker shades from the other end of the grayscale spectrum. If the range of the input values lie between 0 and 255, then the output will also lie within the same range. If you aren't convinced, here is some mathematical proof for you:

(Assuming the input pixels are from an 8-bit grayscale color space)

(Multiplying by-1)

(Adding 255)

Some books prefer to express the negative transformation as follows:

s=T(r)=(N-1-r)

Here, N is the number of grayscale levels in the color space we are dealing with. We have decided to stick with the former definition because all the color spaces we'll be dealing with in this book will involve values between 0 and 255. Hence, we can avoid the unnecessary fastidiousness.

The implementation of a lookup table for the negative transform is also fairly straightforward. We just replace the i with (255-i):

vector<uchar> getNegativeLUT() { 
  vector<uchar> LUT(256, 0); 
  for (int i = 0; i < 256; ++i) 
    LUT[i] = (uchar)(255 - i); 
  return LUT; 
} 

Let's run the code for negative transformation on some images and check what kind of an effect it produces:

The preceding image serves as our input image and the output (negative transformed image) is as follows:.

You can notice the darker pixels that make up the woman's hair (and feathers on the cap) have been transformed to white, and so have the eyes. The shoulders, which were on the fairer side of the spectrum, have been turned darker by the transform. You can now start to appreciate the kind of visual changes that this conceptually simple transformation seems to bring about in images! We iterate, once again, that the image manipulation apps, at the most basic, operate on similar principles.

If you are wondering who the lady in the image is, she goes by the name, Lena. Rather surprisingly, Lena's photograph with her iconic pose has become the de facto standard in the image processing community. A lot of literature in the field of image processing and computer vision uses this photo as an example to demonstrate the workings of some algorithm or technique. So, this is not the first time you'll be seeing her in this book!

Before we finish the section on linear transformations, there is one more thought that I would like to leave you with. You might find some texts out there that like to visualize these transformations graphically as a plot between the input and output pixels. The reason that such visualization is made is because not all linear transformations are as simple as the ones discussed here. We will briefly discuss piecewise linear transformations, where a graphical plot provides a convenient medium to analyze the transformation. But, before that, you will find such a plot for both the identity and the negative transformation drawn on the same graph:

Both the linear transformations that we have discussed so far--identity and negative--are fairly trivial ones. You can have much more complicated forms for s=T(r). For example, instead of the transformation function being linear throughout the entire domain (0 to 255), we can make it piecewise linear. That would mean splitting the input domain into multiple, contiguous ranges and defining a linear transformation for each range, something along the lines of this:

If you are wondering what purpose such a transformation achieves, the answer to that is contrast enhancement. When we say that an image has poor contrast, we mean that some (or all) parts of the image are not clearly distinguishable from their surroundings and neighbors. This happens when the pixels that make up the part of the image all belong to a very narrow band of intensity values. In the graph, consider the portion of the input intensity values that lie around L/2. You will notice that the narrow range of input pixels are mapped to a much wider range of output intensity values. This has been made possible by the steep line (greater slope), which defines the linear transformation around that region. As a result, all pixels that have intensity values within that range will get mapped to the wider output range, thereby improving the contrast of the image.

Now, astute readers might have realized that the shape of the piecewise linear transformation is dependent on the position of the points  and  So, how do we decide the location of the two points? Unfortunately, there is no single, specific answer to this question. As you will learn throughout the course of this book, in most cases, there can be no global correct answer for the selection of such parameters in image processing or computer vision. It depends on the kind of data (within the domain of computer vision, data often equates to images) that we are given to work with.

It would be a good exercise to try and implement the lookup table for a piecewise linear transformation. You can control the shape of the curve by varying the position of the two points and try to see what kind of an effect it has on the algorithm performance.

Simply put, the log transform takes the (scaled) logarithm of every input pixel intensity value. Let's put it down in terms of a mathematical equation:

First, note that the input intensity values have all been incremented by 1 (r+1). This is because our input values vary from 0 to 255 and the logarithm of 0 is not defined. Secondly, there has been no mention regarding the base of the logarithm. Although conceptually the value of the base doesn't really matter (as long as it's kept same throughout the computation), for all practical purposes, we will assume it to be 10. So, when we write log, we actually mean  . Thirdly, you must be wondering about the constant c in the formula. What's it doing there? To answer that question, we need to know the range of output values for log(r+1) when . To help us, I have plotted a graph of the function log(r+1):

As r varies from 0 to 255, log(r+1) ranges from 0 to 2.4. It's in the nature of a logarithmic function to compress the range of input data, as is evident here: an input range spanning 256 values has been compressed to a mere range of 2.4. Does this mean that the output image will have a grayscale range of merely two or three values? It had better not, otherwise, the only thing you'll be able to see is complete darkness! This is where the multiplicative constant c comes into the picture. The role of the multiplier is to make the log-transformed pixel values span the entire range of 256 grayscale levels available for the output image. The way it's done is by choosing a value of c so that the maximum intensity available in the input image gets mapped to 255 in the output. This means that , further implies . Often, for sufficiently large and contrast-rich images, it so happens that the maximum intensity in the input image is 255, that is, . In such cases, the value of the multiplier is c=105.886.

So far, we have been treating the log transformation in a highly mathematical context. Let's see what happens if we actually apply it to images. The image is made up of two horizontal bands. The first band depicts the grayscale color space from 0 (black) on the left and all the way up to 255 (white) on the right end of the spectrum:

The next band depicts the log transform of the corresponding grayscale values (again, from 0 to 255, as we move from left to right). A comparison between the two should give you an idea of what the log transform does to the grayscale spectrum. A glance will tell you that the log-transformed band is much more brighter than its counterpart. Why does this happen?

To give you a better perspective, intensity values of 0 and 15 in the input are mapped to 0 and 127 in the output. This means that if there are two adjacent pixels with intensities 0 and 15 in the input image, both of them would be almost indistinguishable. Human eyes will not be able to perceive such a subtle change in the grayscale intensity. However, in the log-transformed image, the pixel with the intensity value of 15 gets converted to 127 (which lies in the middle of the grayscale spectrum). This would render it clearly distinguishable from its neighbor, which is still completely black!

The exact opposite phenomenon takes place at the other end of the spectrum. For example, pixels with intensities of 205 and 255 are mapped to 245 and 255 by the log transform. This means that a significant difference of 50 in the grayscale spectrum has been reduced to a mere gap of 10. So, the log transform essentially magnifies the differences in intensity of pixels in the lower (darker) end of the grayscale spectrum at the cost of diminishing differences at the higher (brighter) end (notice the steepness of the log curve in the beginning and how it flattens as it reaches the end). In other words, the log transform will magnify details (by enhancing contrast) in the darker ends of the spectrum at the cost of decreasing the information content held by the higher end of the spectrum.

Now that you have an idea of the kind of changes brought forth in grayscale by a log transform, it's time we take a look at some real examples. If you have ever used a camera, you would know that pictures, when taken against a source of light (such as the sun or an artificial source such as a bulb or a tube-light) appear darker. The following image is an example of an image taken under such lighting conditions:

Now try to think of what would happen if we applied the log transform to such an image. We know that a log transform would enhance details from the darker region at the cost of information from the brighter regions of the image. The log-transformed image is shown next. We can see that the darker regions in our original image such as the face and the back of the chair in the background have been rendered more rich in contrast. On the other hand, there has been a significant loss in detail from the brighter segments, such as the table behind the person. This proves that the log transform can be effective in editing pictures that have been captured against the light source by digging out contrast information from the darker regions of an image at the cost of the brighter segments

Before we move on to the implementation, let's see one more application where a log transform may be considered useful. There are some scientific disciplines where we might come across patterns such as the one depicted in the following image:

This image represents a pattern made by a light source on a dark background. More specifically, this is the representation of the Fourier transform of an image. As you can see, there definitely seems to be a pattern, but it's not clearly visible in its native form. We need a way to magnify and enhance these variations that are too subtle to be detected by the naked eye. Log transform to the rescue once more!

The log-transformed image is shown adjacent to the original one. We can observe the pattern quite clearly here:

Now that we have familiarized ourselves with the mathematics behind the log transform and seen it operate on and transform images, we come to the most exciting part where we attempt to mimic their behavior via our OpenCV/C++ code. In accordance with the protocol we have adhered to so far, we first show the code that generates a lookup table for the log transform:

#include <cmath> 
 
vector<uchar> getLogLUT(uchar maxValue) { 
  double C = 255 / log10(1 + maxValue); 
   
vector<uchar> LUT(256, 0); 
  for (int i = 0; i < 256; ++i) 
    LUT[i] = (int) round(C * log10(1+i)); 
  return LUT; 
} 

We notice that the lookup table function is a bit different and slightly more involved than the ones we have discussed thus far. This is mainly because it requires a parameter to operate upon the maximum pixel intensity value in the input image. Recall the description of the log transform, where we discussed that the value of the multiplicative constant c is calculated on the basis of the maximum intensity value, , among the input pixels--. Knowing this fact, the remainder of the function is similar in structure to what we have seen so far.

Now, since the function that returns our lookup table (getLogLUT()) requires an additional parameter, we would have to make appropriate changes to the code that makes calls to it, that is, our processImage() method. The code for our processImage() method is as follows:

void processImage(Mat& I) { 
  double maxVal; 
  minMaxLoc(inputImage, NULL, &maxVal); 
  vector<uchar> LUT = getLUT((uchar) maxVal); 
   
  for (int i = 0; i < I.rows; ++i) { 
    for (int j = 0; j < I.cols; ++j) 
      I.at<uchar>(i, j) = LUT[I.at<uchar>(i, j)]; 
  } 
} 

The one thing that is noteworthy in the preceding snippet is the use of a method named minMaxLoc(). As per the documentation, the function is used to find the minimum and maximum element values and their positions within the array (and by array here, we are referring to a Mat object). The first argument is, of course, the name of the Mat object. The second and the third arguments are the pointers to the minimum and maximum elements, as computed by the function. We have passed the second argument as null because we aren't really interested in the minimum value for now. Apart from the call to minMaxLoc(), the structure of the remainder of processImage() should be familiar to you.

The implementation technique that we have employed for implementing the log transform has followed the framework that we established early on: lookup tables and image traversals. However, as we progress through this book, you will come to appreciate the fact that often, there are multiple ways to reach the same endpoint while implementing your programs in OpenCV. Although this is true for programming in general, we want to focus on how OpenCV provides us with options that allow us to perform and (more often than not) simplify tasks that otherwise would take a lot of tedious steps (iterations) to achieve. To that end, we will present another technique using OpenCV to compute the log transformation for images.

Like always, we first begin by including the relevant headers and namespaces:

#include <opencv2/core/core.hpp> 
#include <opencv2/highgui/highgui.hpp> 
#include <opencv2/imgproc/imgproc.hpp> 
 
using namespace std; 
using namespace cv; 

Barring the declarations, our code that initially spanned a couple of user-defined functions and a main() class has now been essentially reduced to five lines of code that do all the work! Nowhere do we explicitly traverse any data matrix to modify pixel values based on some predefined transformation functions. The native methods that we use do that in the background for us. Have a look at the following code:

int main() { 
  Mat input_image = imread("lena.jpg", IMREAD_GRAYSCALE); 
  Mat processed_image; 
   
  input_image.convertTo(processed_image, CV_32F); 
  processed_image = processed_image + 1; 
  log(processed_image, processed_image); 
  normalize(processed_image, processed_image, 0, 255, NORM_MINIMAX); 
  convertScaleAbs(processed_image, processed_image); 
 
  imshow("Input image", image); 
  imshow("Processed Image", processed_image); 
  waitKey(0); 
   
  return 0; 
} 

The five major functions used have been described in detail as follows:

One of the most prominent and striking differences that you will notice with this method is that it completely relies on the functions provided by the OpenCV API. What we have essentially done is avoid reinventing the wheel. Knowing how to traverse data matrices was, no doubt, an important skill to master. However, something as basic as iterating Mat objects becomes tedious, time consuming, and off-topic when we have big and complex computer vision systems to build. In such scenarios, it is good to utilize the features of the library if they have been made available to us. A classic example is the overloading of mathematical operators for the Mat class. Imagine if we had to implement a fully-fledged matrix traversal every single time we needed an operation as simple as increment all pixels by 1. To keep things concise and readable in our code and speed up the development cycle at the same time, the library has afforded us the luxury of writing I=I+1, even for the objects of the Mat class! Another advantage that we get if we rely on the OpenCV functions as much as possible is that we are guaranteed that the code that runs is heavily optimized and efficient in terms of memory and runtime.

The developers at OpenCV have built as many abstractions over such behind-the-scenes, plumbing operations as is required by programmers like us to seamlessly develop a varied set of applications that falls within the domain of computer vision and machine learning, without having to worry about the intricacies of implementation. This will be a recurrent theme in our book across most of the chapters.

Before we finish this section, we will visit our final transformation that goes by the name of exponential transform. What it does essentially is the complete opposite of the log transform (hence, it is also named inverse-log transform). While the log transform enhanced the pixels in the lower end of the spectrum, the exponential transform does the same for the pixels at the high intensity end of the spectrum. Mathematically, we have the following:

Just like computing the log operator essentially involves taking the logarithm of the intensity values of every input pixel, the exponential transform raises a base value b to the power of the input pixel's intensity value. We subtract  so that when the input is 0, the output gets mapped to 0 as well. The constant c plays the same role as in the case of log transform, ensuring that the output lies in the range of 0 to 255. The value of the constant b decides the shape of the transform. Typically, b is chosen to lie close to 1. The following graph depicts a plot of both the log and the exponential transform (b=1.02):

The shape of the plots brings out the complementary nature of both the transforms. On one hand, the log transform maps a narrow range of input intensity values at the lower end of the grayscale spectrum to a broader range at the output. On the other hand, the curve of the exponential transform becomes steep at the other end of the spectrum, thereby mapping a narrow range of input values to a much larger range at the output. To further illustrate the dichotomy between the two, the following figure demonstrates the changes that the exponential transform does to a grayscale spectrum. This is similar to the kind of grayscale comparisons that we did for the Log transformation. The following image depicts the original grayscale spectrum from 0 (on the left) all the way to 255 (on the right):

You'll find the corresponding spectrum for exponential transform in the following image. Note how the entire spectrum is darker (as opposed to lighter, in the case of log transforms) than the original grayscale band. Using the same line of reasoning that we presented in the section on log transforms, you can deduce why that happens:

As always, we share the code to compute the lookup table for the exponential transform:

const int BASE = 1.02; 
vector<uchar> getExpLUT(uchar maxValue) { 
  double C = 255.0 / (pow(BASE, maxValue) - 1); 
   
vector<uchar> LUT(256, 0); 
  for (int i = 0; i < 256; ++i) 
    LUT[i] = (int) round(C * (pow(BASE, i) - 1)); 
  return LUT; 
} 

The code is exactly similar to the one for calculating log transforms except for the formula. We won't be going over the other functions, such as the traversal and the main() function once again. I would strongly suggest you to implement the exponential transform by using the OpenCV functions (and avoid reinventing the wheel by implementing matrix traversals) as we did in the case of log transforms. Take the help of online documentation (OpenCV has excellent online documentation) to find the function that would enable you to take the exponential of pixel values. This is an important skill to learn. There are so many different functions spread across different modules within OpenCV and the documentation is the only reliable and up-to-date source of information. As you go on to develop bigger and more powerful applications, the documentation will be your only ally to help you navigate your way through all the different functions.

Also, we show an example of how the exponential transformation works on images. The following is our original input image:

Applying an exponential transform leads us to the following:

The overall darkening of the input image is quite apparent!

We have discussed the advantages of using a lookup table-based approach for implementing grayscale transformations. In fact, we have also been implementing all our transformations using a framework based on a combination of computing the lookup table and traversing the data matrix. If this particular combination is so efficient as well as ubiquitous, haven't the OpenCV developers thought of implementing this for us already? If you've followed the trend of this chapter, you would've guessed the answer to the question by now! Yes, OpenCV does have a function that allows you to do exactly that: provide it with a lookup table and a Mat object, and it will transform each pixel of the Mat object on the basis of the rules laid down by the lookup table, and store the result in a new Mat object. What's even better is that the function is named LUT()! Let's look at a sample code snippet that uses the LUT() method to implement the negative transform.

As we hinted just now, the LUT() method requires three parameters:

We have been dealing with the first and the third throughout the chapter. How do we pass the lookup table to the LUT() method? Remember that a lookup table is essentially an array (or a vector). We have been treating it as such in all our implementations so far, and we also know that the Mat class in OpenCV is more than equipped to handle the processing of one-dimensional arrays. Hence, we would be passing our lookup table as another Mat object. Since our LUT is essentially a Mat object, we change our getLUT() function as follows:

Mat getNegativeLUT() { 
vector<uchar> lut_array(256, 0); 
for (int i = 0; i < 256; ++i) 
  lut_array[i] = (uchar)(255 - i); 
 
Mat LUT(1, 256, CV_8U); 
for (int j = 0; j < 256; ++j) 
  LUT.at<uchar>(0, j) = lut_array[j]; 
return LUT; 
} 

Notice that the first three lines are identical to what we have been doing so far-initializing and constructing our lookup table as a C++ vector. Now, we take that vector and transform it into a Mat object having one row and 256 columns and type CV_8U (which makes it the perfect container for the elements of a C++ vector of uchar). The remainder of the function makes that transition and returns the Mat object as our LUT.

Once the LUT has been created, applying it is as simple as calling OpenCV's LUT() method with all the necessary arguments:

LUT(input_image, lookup_table, output_image);