How to quantify data into business value? It's a serious question that we might be prompted to ask when we take a look around and notice the increasing appetite of users for rich media and the content of data across the web. That could generate several challenging points: How to manage the exponential amount of data? Particularly, how to extract from these immense waves of data the most valuable aspects? It is the era of big data! To meet the growing demand of big data and facilitate its analysis, few solutions such as Hadoop and Spark appeared and have become a necessary tool towards making a first successful step into the big data world. However, the first question was not sufficiently answered! It might be needed to introduce a new architecture and cost approach to respond to the scalability of intensive resources consumed when analyzing data. Although Hadoop, for example, is a great solution to run data analysis and processing, there are difficulties with configuration and maintenance. Besides, its complex architecture might require a lot of expertise. In this book, you will learn how to use OpenStack to manage and rapidly configure a Hadoop/Spark cluster. Sahara, the new OpenStack integrated project, offers an elegant self-service to deploy and manage big data clusters. It began as an Apache 2.0 project and now Sahara has joined the OpenStack ecosystem to provide a fast way of provisioning Hadoop clusters in the cloud. In this chapter, we will explore the following points:
Introduce briefly the big data groove
Understand the success of big data processing when it is combined with the cloud computing paradigm
Learn how OpenStack can offer a unique big data management solution
Discover Sahara in OpenStack and cover briefly the overall architecture
A world of information, sitting everywhere, in different formats and locations, generates a crucial question: where is my data?
During the last decade, most companies and organizations have started to realize the increasing rate of data generated every moment and have begun to switch to a more sophisticated way of handling the growing amount of information. Performing a given customer-business relationship in any organization depends strictly on answers found in their documents and files sitting on their hard drives. It is even wider, with data generating more data, where there comes the need to extract from it particular data elements. Therefore, the filtered elements will be stored separately for a better information management process, and will join the data space. We are talking about terabytes and petabytes of structured and unstructured data: that is the essence of big data.
Big data refers to the data that overrides the scope of traditional data tools to manage and manipulate them.
Gartner analyst Doug Laney described big data in a research publication in 2001 in what is known as the 3Vs:
Note
To read more about the 3Vs concept introduced by Doug Laney, check the following link: http://blogs.gartner.com/doug-laney/files/2012/01/ad949-3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf
Another important question is how will the data be manipulated and managed in a big space? For sure, traditional tools might need to be revisited to meet the large volume of data. In fact, loading and analyzing them in a traditional database means the database might become overwhelmed by the unstoppable massive surge of data.
Additionally, it is not only the volume of data that presents a challenge but also time and cost. Merging big data by using traditional tools might be too expensive, and the time taken to access data can be infinite. From a latency perspective, users need to run a query and get a response in a reasonable time. A different approach exists to meet those challenges: Hadoop.
Hadoop tools come to the rescue and answer a few challenging questions raised by big data. How can you store and manage a mixture of structured and unstructured data sitting across a vast storage network? How can given information be accessed quickly? How can you control the big data system in an enhanced scalable and flexible fashion?
The Hadoop framework lets data volumes increase while controlling the processing time. Without diving into the Hadoop technology stack, which is out of the scope of this book, it might be important to examine a few tools available under the umbrella of the Hadoop project and within its ecosystem:
Apache Spark is another amazing alternative to process large amounts of data that a typical MapReduce cannot provide. Typically, Spark can run on top of Hadoop or standalone. Hadoop uses HDFS as its default file system. It is designed as a distributed file system that provides a high throughput access to application data.
The big data tools (Hadoop/Spark) sound very promising. On the other hand, while launching a project on a terabyte-scale, it might go quickly into a petabyte-scale. A traditional solution is found by adding more clusters. However, operational teams may face more difficulties with manual deployment, change management and most importantly, performance scaling. Ideally, when actively working on a live production setup, users should not experience any sort of service disruption. Adding then an elasticity flavor to the Hadoop infrastructure in a scalable way is imperative. How can you achieve this? An innovative idea is using the cloud.
Note
Some of the most recent functional programming languages are Scala and R. Scala can be used to develop applications that interact with Hadoop and Spark. R language has become very popular for data analysis, data processing, and descriptive statistics. Integration of Hadoop with R is ongoing; RHadoop is one of the R open source projects that exposes a rich collection of packages to help the analysis of data with Hadoop. To read more about RHadoop, visit the official GitHub project page found at https://github.com/RevolutionAnalytics/RHadoop/wiki
Cloud computing technology might be a satisfactory solution by eliminating large upfront IT investments. A scalable approach is essential to let businesses easily scale out infrastructure. This can be simple by putting the application in the cloud and letting the provider supports and resolves the big data management scalability problem.
One shining example is the popular Amazon service named Elastic MapReduce (EMR), which can be found at https://aws.amazon.com/elasticmapreduce/. Amazon EMR in a nutshell is Hadoop in the cloud. Before taking a step further and seeing briefly how such technology works, it might be essential to check where EMR sits in Amazon from an architectural level.
Basically, Amazon offers the famous EC2 service (which stands for Elastic Compute Cloud) that can be found at https://aws.amazon.com/ec2/. It's a way that you can demand a certain size of computations resources, servers, load balancers, and many more. Moreover, Amazon exposes a simple key/value storage model named Simple Storage Service (S3) that can be found at https://aws.amazon.com/s3/.
Using S3, storing any type of data is very simple and straightforward using web or command-line interfaces. It is the responsibility of Amazon to take care of the scaling, data availability, and the reliability of the storage service.
We have used a few acronyms: EC2, S3 and EMR. From high-level architecture, EMR sits on top of EC2 and S3. It uses EC2 for processing and S3 for storage. The main purpose of EMR is to process data in the cloud without managing your own infrastructure. As described briefly in the following diagram, data is being pulled from S3 and is going to automatically spin up an EC2 cluster within a certain size. The results will be piped back to S3. The hallmark of Hadoop in the cloud is zero touch infrastructure. What you need to do is just specify what kind of job you intend to run, the location of the data, and from where to pick up the results.
OpenStack is a very promising open source cloud computing solution that does not stop adumbrating and joining different projects related to the cloud environment. OpenStack kept growing its ecosystem thanks to the conglomeration of many projects that make it a very rich cloud platform. OpenStack exposes several infrastructure management services that work in tandem to provide a complete suite of infrastructure management software. Most of its modules have been refined and become more mature within the Havana release. It might be essential first to itemize the most basic ones briefly:
Keystone: The identity management service. Connecting and using OpenStack services requires in the first place authentication.
Glance: The image management service. Instances will be launched from disk images that glance stores them in its image catalogue.
Nova: The instance management service. Once authenticated, a user can create an instance by defining basic resources such as image and network.
Cinder: The block storage management service. It allows creating and attaching volumes to instances. It also handles snapshots, which can be used as a boot source.
Neutron: The network management service. It allows creating and managing an isolated virtual network for each tenant in an OpenStack deployment.
Swift: The object storage management service. Any form of data in Swift is stored in a redundant, scalable, distributed object storage using a cluster of servers.
Heat: The orchestration service. It provides a fast-paced way to launch a complete stack from one single template file.
Ceilometer: The telemetry service. It monitors the cluster resources used in an OpenStack deployment.
Horizon: The OpenStack Dashboard. It provides a web-based interface to different OpenStack services such as Keystone, Glance, Nova, Cinder, Neutron, Swift, Heat, and so on.
Trove: The Database as a Service (DBaaS) component in OpenStack. It enables users to consume relational and non-relational database engines on top of OpenStack.
Note
At the time of writing, more incubated projects are being integrated in the OpenStack ecosystem with the Liberty release such as Ironic, Zaqar, Manilla, Designate, Barbican, Murano, Magnum, Kolla, and Congress. To read more about those projects, refer to the official OpenStack website at: https://www.openstack.org/software/project-navigator/
The awesomeness of OpenStack comes not only from its modular architecture but also the contribution of its large community by developing and integrating a new project in nearly every new OpenStack release. Within the Icehouse release, OpenStack contributors turned on the light to meet the big data world: the Elastic Data Processing service. That becomes even more amazing to see a cloud service similar to EMR in Amazon running by OpenStack.
Well, it is time to open the curtains and explore the marriage of one of the most popular big data programs, Hadoop, with one of the most successful cloud operating system OpenStack: Sahara. As shown in the next diagram of the OpenStack IaaS (short for Infrastructure as a Service) layering schema, Sahara can be expressed as an optional service that sits on top of the base components of OpenStack. It can be enabled or activated when running a private cloud based on OpenStack.
Note
More details on Sahara integration in a running OpenStack environment will be discussed in Chapter 2, Integrating OpenStack Sahara.
Sahara is an incubated project for big data processing since the OpenStack Icehouse release. It has been integrated since the OpenStack Juno release. The Sahara project was a joint effort and contribution between Mirantis, a major OpenStack integration company, Red Hat, and Hortonworks. The Sahara project enables users to run Hadoop/Spark big data applications on top of OpenStack.
The main reason the Sahara project was born is the need for agile access to big data. By moving big data to the cloud, we can capture many benefits for the user experience in this case:
Unlimited scalability: Sahara sits on top of the OpenStack Cloud management platform. By its nature, OpenStack services scale very well. As we will see, Sahara lets Hadoop clusters scale on OpenStack.
Elasticity: Growing or shrinking, as required, a Hadoop cluster is obviously a major advantage of using Sahara.
Data availability: Sahara is tightly integrated with core OpenStack services as we will see later. Swift presents a real cloud storage solution and can be used by Hadoop clusters for data source storage. It is a highly durable and available option when considering the input/output of processing a data workflow.
Note
Swift can be used for input and output data source access in a Hadoop cluster for all job types except Hive.
For an intimate understanding of the benefits cited previously, it might be essential to go through a concise architectural overview of Sahara in OpenStack. As depicted in the next diagram, a user can access and manage big data resources from the Horizon web UI or the OpenStack command-line interface. To use any service in OpenStack, it is required to authenticate against the Keystone service. It also applies to Sahara, which it needs to be registered with the Keystone service catalogue.
To be able to create a Hadoop cluster, Sahara will need to retrieve and register virtual machine images in its own image registry by contacting Glance. Nova is also another essential OpenStack core component to provision and launch virtual machines for the Hadoop cluster. Additionally, Heat can be used by Sahara in order to automate the deployment of a Hadoop cluster, which will be covered in a later chapter.
Note
In OpenStack within the Juno release, it is possible to instruct Sahara to use block storage as nodes backend.
In addition to sharing the aforementioned generic big data in OpenStack, OpenStack Sahara has some unique characteristics that can be itemized as the following:
Fast provisioning: Deploying a Hadoop/Spark cluster becomes an easy task by performing a few push-button clicks or via command line interface.
Centralized management: Controlling and monitoring a Hadoop/Spark cluster from one single management interface efficiently.
Cluster management: Sahara offers an amazing templating mechanism. Starting, stopping, scaling, shaping, and resizing actions may form the life cycle of a Hadoop/Spark cluster ecosystem. Performing such a life cycle in a repeatable way can be simplified by using a template in which will be defined the Hadoop configuration. All the proper cluster node setup details just get out of the way of the user.
Workload management: This is another key feature of Sahara. It basically defines the Elastic Data Processing, the running and queuing jobs, and how they should work in the cluster. Several types of jobs for data processing such as MapReduce job, Pig script, Oozie, JAR file, and many others should run across a defined cluster. Sahara enables the provisioning of a new ephemeral cluster and terminates it on demand, for example, running the job for some specific analysis and shutting down the cluster when the job is finished. Workload management encloses data sources that defines where the job is going to read data from and write them to.
Note
Data sources URLs into Swift and URLs into HDFS will be discovered in more details in Chapter 5, Discovering Advanced Features with Sahara.
No deep expertise: Administrators and operators will not wonder anymore about managing the infrastructure running underneath the Hadoop/Spark cluster. With Sahara, managing the infrastructure does not require real big data operational expertise.
Multi-framework support: Sahara exposes the possibility to integrate diverse data processing frameworks using provisioning plugins. A user can choose to deploy a specific Hadoop/Spark distribution such as the Hortonworks Data Platform (HDP) plugin via Ambari, Spark, Vanilla, MapR Distribution, and Cloudera plugins.
Analytics as a Service: Bursty analytics workloads can utilize free computing infrastructure capacity for a limited period of time.
We have seen in the previous diagram how Sahara has been integrated in the OpenStack ecosystem from a high-level perspective. As it is a new OpenStack service, Sahara exposes different components that interact as the client of other OpenStack services such as Keystone, Swift, Nova, Neutron, Glance, and Cinder. Every request initiated from the Sahara endpoint is performed on the OpenStack services public APIs. For this reason, it is essential to put under scope the Sahara architecture as shown in the following diagram:
The OpenStack Sahara architecture consists essentially of the following components:
REST API: Every client request initiated from the dashboard will be translated to a REST API call.
Auth: Like any other OpenStack service, Sahara must authenticate against the authentication service Keystone. This also includes client and user authorization to use the Sahara service.
Vendor Plugins: The vendor plugins sit in the middle of the Sahara architecture that exposes the type of cluster to be launched. Vendors such as Cloudera and Apache Ambari provide their distributions in Sahara so users can configure and launch a Hadoop based on their plugin mechanism.
Elastic Data Processing (EDP): Enables the running of jobs on an existing and launched Hadoop or Spark cluster in Sahara. EDP makes sure that jobs are scheduled to the clusters and maintain the status of jobs, their sources, from where the data sources should be extracted, and to where the output of the treated data sources should be written.
Orchestration Manager/Provisioning Engine: The core component of the Sahara cluster provisioning and management. It instructs the Heat engine (OpenStack orchestrator service) to provision a cluster by communicating with the rest of the OpenStack services including compute, network, block storage, and images services.
Data Access Layer (DAL): Persistent internal Sahara data store.
Note
It is important to note that Sahara was configured to use a direct engine to create instances of the cluster which initiate calls to the required OpenStack services to provision the instances. It is also important to note that Direct Engine in Sahara will be deprecated from OpenStack Liberty release where Heat becomes the default Sahara provisioning engine.
In this chapter, you explored the factors behind the success of the emerging technology of data processing and analysis using cloud computing technology. You learned how OpenStack can be a great opportunity to offer the needed scalable and elastic big data on-demand infrastructure. It can be also useful to execute on-demand Elastic Data Processing tasks.
The first chapter exposed the new OpenStack incubated project called Sahara: a rapid, auto-deploy, and scalable solution for Hadoop and Spark clusters. An overall view of the Sahara architecture has been discussed for a fast-paced understanding of the platform and how it works in an OpenStack private cloud environment.
Now it is time to get things running and discover how such an amazing big data management solution can be used by installing OpenStack and integrating Sahara, which will be the topic of the next chapter.