Effective Business Intelligence with QuickSight

We live in a digital era where data is generated everywhere, from smart connected devices to social media. In 2014, every second over 5,700 tweets were sent and 800 links were shared using Facebook, and the digital universe expanded by about 1.7 MB per minute for every person on earth (source: IDC 2014 report). This amount of data sharing and storing is unprecedented and is contributing to what is known as big data. In 2013, about 4.4 ZB were created and in 2020 the forecast is 44 ZB, which is 44 trillion GB (source: http://www.emc.com/leadership/digital-universe/2014iview/executive-summary.htm).

Figure 1.1: Data growth

IDC predicts that organizations that are able to analyze this big data and derive actionable insights will see an additional $430 billion in productivity benefits over their peers (source: IDC FutureScape: Worldwide Big Data and Analytics 2016 Predictions, https://www.cloudera.com/content/dam/www/static/documents/analyst-reports/idc-futurescape.pdf).

Let's look at some real use cases that have benefited from big data:

This unprecedented growth of data has resulted in need for faster insights, quicker co-relations, and the need to democratize data and analysis.  Let's next look at the current BI landscape, key features they provide, and also their limitations.

Over the last 20 years, several vendors have built products to satisfy BI needs for organizations and this space is pretty crowded with big vendors and niche players that include Tableau desktop, Oracle OBIEE, SAP Business Objects, IBM Cognos, Qlick, and DOMO. Let's review the key features provided by the BI vendors.

Let's review the process for creating insights using traditional BI tools like Oracle OBIEE, SAP Business Objects, and IBM Cognos. At a high level, building a BI dashboard involves the following:

The preceding data flow is shown in the following diagram and is primarily built by IT with regular consultation with data stewards and dashboard consumers:

Figure 1.2: Process flow for traditional BI tools

There is an explosion of data being generated by cloud applications like Salesforce.com, Adobe Analytics, and HubSpot. In addition, enterprises are increasingly migrating their data to cloud with services like Dropbox, DocuSign, Google, AWS S3, and AWS RDS. 

While enterprises have adopted Software as a Service (SaaS) for applications; the move to BI as a service has been slower. This trend is about to change and IDC predicts that by 2020, 50% of business analytics software will incorporate predictive analytics based on cloud platforms. 

The definition of cloud BI would vary based on whom you talk to. First you have the traditional enterprise BI vendors; who have ability to host their software on a cloud server and call it a cloud service. Then you have the next generation BI vendors that have designed their software from ground up to use the elastic cloud services of Microsoft, AWS, and Google and offer real cloud BI service. AWS QuickSight falls under the second category of cloud BI vendors. Here are some of the popular cloud BI vendors Microsoft Power BI, Looker, Chartio, and Bime.

There are several operational and financial drivers that work in favor of cloud  BI service, the key ones being:

Next we will look into what makes AWS QuickSight stand out when compared to the others.

The Amazon web services team has been in the forefront of providing real solutions that are designed for massive scale; their S3 storage is the most popular cloud storage service and it is used by companies like Netflix, Nasdaq, Airbnb, and Redfin to name just a few. QuickSight is a new cloud BI service specifically designed to address the need for every day analytics and reporting. QuickSight allows enterprises to get started in minutes, access data from multiple sources, build interactive visuals, get answers fast, and tell a story with data.

While QuickSight has competitors, I believe it is poised to become the leader in cloud powered BI due to the deep connectivity to AWS services, next generation distributed architecture, with an in-memory caching layer on the worlds most scalable infrastructure. Additionally, AWS has made it extremely affordable that competitors will find it hard to match. Let's review in detail how QuickSight differs from other BI tools.

Amazon QuickSight was built to address pain points of the traditional BI tools and provides IT and business teams with a fast, cloud-powered BI service at one-tenth the cost of traditional BI software.

Here are the key features that make QuickSight the next generation cloud-powered BI tool:

The following diagram shows the simplified flow from data to visuals using QuickSight:

Figure 1.3: QuickSight simplified process

It has the following four steps, most of which can be done by a power user without the need for personal IT:

Let's take a deeper look into how the complete solution architecture will look with QuickSight. The following diagram shows the high level architecture that takes data from various sources and presents as insights using QuickSight:

Figure 1.4: QuickSight architecture

Let's review this architecture starting from the bottom and going to the top:

Next, let's look at how easy it is to get started and build a dashboard using QuickSight.

In this section, we will review how to get started with QuickSight and build a real dashboard on data.

{
  "Statement": [
    {
      "Action": [
        "iam:ListPolicyVersions",
        "iam:ListAccountAliases",
        "iam:AttachRolePolicy",
        "iam:GetPolicy",
        "iam:GetPolicyVersion",
        "iam:CreateRole",
        "iam:CreatePolicy",
        "iam:CreatePolicyVersion",
        "iam:DeletePolicyVersion",
        "iam:GetRole",
        "iam:ListAttachedRolePolicies",
        "iam:ListRoles",
        "ds:CheckAlias",
        "ds:CreateIdentityPoolDirectory",
        "ds:CreateAlias",
        "ds:AuthorizeApplication",
        "ds:DescribeDirectories",
        "ds:UnauthorizeApplication",
        "ds:DeleteDirectory",
        "quicksight:Subscribe",
        "quicksight:Unsubscribe",
        "s3:ListAllMyBuckets"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

With QuickSight, it is really easy to build your analysis with minimal effort. Let's do a test drive of QuickSight and build our first analysis using data from the US Department of Education that provides information about college tuition across all the states in the USA.

Downloading data

The dataset is available from the following public URL: https://catalog.data.gov/dataset/college-scorecard. Click on the Download icon as shown in the following screenshot to download raw data:

Figure 1.5: College scores raw data

Starting your visualizations

Now you are ready to start visualizing data using the built-in charts in QuickSight. Let's see how to create our first useful chart, which is also demonstrated in the next screenshot. Follow the steps to create a chart showing the average tuition fees by state:

1. First select the horizontal bar chart from the Visual types.

2. Next select the STABBR as the Y axis and TUITFTE as the Value field.

3. Next in the Field wells option, change the Aggregate type of the value from the default Sum to Average.

The visualization is complete, as shown in the following screenshot, and now you can explore the chart and get more insights from the data:

Figure 1.10: Bar chart

Building multiple visualizations

QuickSight additionally supports line graphs, area line charts, scatter plots, heat maps, pie graphs, tree maps, and pivot tables. You can next add another visual for the same dataset pretty easily and get further understanding of the data. In the following steps, we will see how to build a pie chart that shows the sum of in-state tuition by city:

1. Click on the + to add a new visual.

2. Select the pie chart from the Visual types.

3. Next select the CITY as the Group/Color and TUTIONFEE_IN as the Value field.

4. Notice the default aggregation for value is automatically set to Sum.

The visualization is complete, as shown in the following screenshot, and now you can explore the chart and get more insights from the data:

Figure 1.11: Pie chart

We live and breathe data with every action in our life, and organizations that have the ability to analyze relevant data in a timely manner will save an additional $430 billion by 2020 over their peers. While there is a need, the traditional BI platforms are slow, expensive, and are not designed to handle the volume, variety, and velocity needs that organizations face today.

Amazon QuickSight is the innovative and next-generation, cloud-hosted BI platform that addresses short falls of traditional BI systems and is at an extremely competitive price from $9 per user per month. QuickSight can source data from various sources including relational databases, files, streaming, and NoSQL databases. QuickSight also comes with an in-memory caching layer that can cache and calculate aggregates on the fly. With QuickSight, data analysts are truly empowered and can build intuitive reports in 60 seconds without any significant setup by IT. In the next chapter, we will look into the details of onboarding various data sources that are supported by QuickSight.