File formats

Files have been in data processing for decades. A file acts as a persistent storage for data, and programmers have been using files for exchanging or storing data  for a long time now. There are formats associated with each file that tell you how data is being written and read from the file. For example, if we consider the .csv file, we may assume that each record is separated by a line and each column is separated by a delimiter. If the writer does not put data in the specific agreed format, then the reader may read data wrongly or processing logic may break. Each file format has a specific way of storing the data and there are different parameters that can help you decide which file format is best suited for your application. In this section, we will walk you through some of the widely used file formats and their internals. 

Understanding file formats

Before we dive into the details of different big data formats, we need to establish a method that we can use to evaluate when...