Search icon CANCEL
Cart icon
Close icon
You have no products in your basket yet
Save more on your purchases!
Savings automatically calculated. No voucher code required
Arrow left icon
All Products
Best Sellers
New Releases
Learning Hub
Free Learning
Arrow right icon
Analytics for the Internet of Things (IoT)
Analytics for the Internet of Things (IoT)

Analytics for the Internet of Things (IoT): Intelligent analytics for your intelligent devices

By Andrew Minteer
$39.99 $27.98
Book Jul 2017 378 pages 1st Edition
$39.99 $27.98
$15.99 Monthly
$39.99 $27.98
$15.99 Monthly

What do you get with eBook?

Product feature icon Instant access to your Digital eBook purchase
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
Buy Now
Table of content icon View table of contents Preview book icon Preview Book

Analytics for the Internet of Things (IoT)

Chapter 1. Defining IoT Analytics and Challenges

In this chapter, we will discuss some concepts and challenges associated with analytics on Internet of Things (IoT) data. We will cover the following topics:

  • The analytics maturity model
  • Defining the IoT
  • What is different about IoT data?
    • Data volume
    • Problems with time
    • Problems with space
    • Data quality
    • Analytics challenges
  • Concerns with finding business value

The situation

The tense white-yellow of the fluorescent ceiling lights press down on you while you sit in your cubicle and stare at the monitors on your desk. You sense it is now night outside but can't see over the fabric walls to know for sure. You stare at the long list of filenames on one screen and the plain text rows of opaque sensor data on the other screen.

Your boss had just left to angrily brood somewhere in the office, and you are not sure where. He had been glowering over your shoulder.

"We spent $20 million in telecommunication and consulting fees last year just to get this data! The hardware costs $20 per unit. We've been getting data, and it has been piling up costing us $10,000 a month. There are 20 TB of files - that's big data, isn't it? And we can't seem to do anything with it?"he had said.

"This is ridiculous!," he continued, "It was supposed to generate $100 million in new revenue. Where is our first dollar? Why can't you do anything with it? I have five consultants a week calling me to tell me they can handle it- they'll even automate it. Maybe we should just pick one and hope they aren't selling us snake oil."

You know he does not really blame you. You were a whiz with Excel and knew how to query databases. A lot of analytics requests went to you. When the CEO decided the company needed a big data guy, they hired a VP out of Silicon Valley. But the new VP ended up taking a position with a different Silicon Valley company the day before he was supposed to start at your company.

You were hastily moved into the new analytics group. A group of one - you. It was to be a temporary shift until another VP was found. That was six months ago. The company is freezing funds for outside training and revenues are looking tight. So, no training for you.

Although many know the terms, no one in the company actually understands what Hadoop is or how to even start using this thing called machine learning. But others more and more seem to expect you to not only know it but already be doing it.

Executives have been reading articles in HBR and Forbes about the huge potential of the IoT combined with Artificial Intelligence or AI. They feel like the company will be left behind, and soon, if it does not have its own IoT big data solution incorporating AI. Your boss is feeling the pressure. Executives have several ideas for him where AI can be used. They seem to think that getting the idea is the hard part, implementation should be easy. Your boss is worried about his job and it rolls downhill to you.

Your screen on the left looks like this:

The list goes on and on for several pages. You have been able to combine several files and do some pivot tables and charting in Excel. But it takes a lot of your time, and you can only realistically handle a month or two worth of data. The questions are coming in faster than your ability to answer them. You and your boss have been talking about bringing in temps to do the work–they don't really need to understand it, just follow the steps that you outline for them.

Your screen on the right looks like this:

Your IT department has been consolidating lots of little files into several very large ones. The filesystem was being overloaded by the number of files, so the solution was to consolidate. Unfortunately, for you, many files are now too large to open in Excel, which limits what you can do with them. You end up doing more analytics on recent data simply because it is much easier (the files are still small).

Looking at the data rows, it is not obvious what you can do with it beyond sums and averages. The files are too big to do a VLOOKUP in Excel against something like your production records - which is stored in files often too big to even open in Excel.

At this point, you can't begin to think how you would apply Machine Learning to this data. You are not quite sure what it even means. You know the data is difficult to manipulate for anything beyond recent datasets. Surely, long periods of time would be needed to extract value out of it.

You hear a cough from behind you. Your boss is back.

He says quietly and stiffly, "I'm sorry. We're going to have to hire a consultant to take this over. I know how hard you've been working. You've done some amazing things considering the limitations, and nobody appreciates that enough. But I have to show results. It will probably take a month or two to fully bring someone on board. In the meantime, just keep at it–maybe we can make a breakthrough before then."

Your heart sinks. You are convinced there is huge value in the connected device data. You feel like you could make a career out of IoT analytics if you could just figure out how to get there. But you are not a quitter.

You decide you will not go down without a fight, you will find a way.

Defining IoT analytics

In order to understand IoT analytics, it is helpful to separate it out and define both analytics and the IoT. This will help frame the discussion for the rest of the book.

Defining analytics

If you ask a hundred people to define analytics, you are likely to get a hundred different answers. Each person tends to have his or her own definition in mind that can range from static reports to advanced deep learning expert systems. All tend to call efforts in the wide ranging territory analytics without much further explanation.

We will take a fairly broad definition in this book as we are covering quite a bit of territory. In their best selling book Competing on Analytics, Tom Davenport and Jeanne Harris created a scale, which they called Analytics Maturity. Companies progress to higher levels in the scale as their use of analytics matures, and they begin to compete with other companies by leveraging it.

When we use the word analytics, we will mean using techniques that fall in the range from query/drill down to optimization as shown in the following chart from Competing on Analytics:

We will also take a slightly different philosophy. Unlike the notion of a company progressing through each level to get to the peak of maturity at the upper right with optimization, we will strive to reach success at all levels in parallel.

The idea of a company not being analytically mature unless it is actively employing optimization models at every turn can be dangerous. This puts pressure on a company to focus time and resources where there may not be a return on investment (ROI) for them. Since resources are always limited, this could also cause them to under-invest in projects in other areas that have a higher ROI.

The reason for the lack of ROI is often that a company simply does not have the right data to take full advantage of the more advanced techniques. This could be no fault of their own as the signal in the noise may be just too weak to tease out. This could stem from the state of technology, not yet at the point where the key predictive data can even be monitored. Or even if this is possible, it may be far too expensive to justify capturing it. We will talk about the limitations of available data quite a bit in this book. The goal will always be to maximize ROI at all levels of the maturity model.

We will also take the view that analytics maturity is about having the capability and knowing how to enable the full scale. It is not about what you are doing. It is about what you are capable of doing in order to maximize your sum total ROI across the full scale. Each level can be exploited if an opportunity is spotted. And we want there to be fertile ground for opportunities across the full scale. More about this will be covered throughout the book.

Defining the Internet of Things

Sensors have been tracking data for decades at manufacturing plants, retail stores, and remote oil and gas equipment. Why all of sudden is there this IoT hype all over the media?

The dramatic decrease in sensor costs, bandwidth costs, the spread of cellular coverage, and the rise of cloud computing all combine to create fertile conditions to easily connect devices over the internet. For example, as shown in the following graph, Goldman Sachs predicts an average sensor cost in 2020 of under $0.40 USD, 30% of what it was in 2004. Whether all these devices should be connected or not is hotly debated:

Data source: Goldman Sachs, BI estimates

The definitions of IoT seem to vary quite a bit; some include machine sensors only, others include RFID tags and smartphones.

We will use this definition from Forrest Stroud on Webopedia:

The IoT refers to the ever-growing network of physical objects that feature an IP address for the internet connectivity and the communication that occurs between these objects and other internet-enabled devices and systems.

Or to get even more basic: stuff that talks to other stuff over the internet without requiring you to do anything. This clears it up, right?

Even the number of things projected to be connected by 2020 varies widely. Some sources project 20.8 billion devices, others project up to 50 billion - over twice the amount.

For our purposes, we are more concerned with how to analyze the data generated than we are about the scope of devices that should be considered part of the IoT. If something sends data remotely by way of the internet, it is fair game for us, especially if it is machine-generated on one end and machine-consumed on the other.

We are more concerned with how to extract value from the data and adapt to circumstances inherent to it. IoT is not really new, as elements of it have been developing for decades. Remote detection of oil well spills was happening in the 1970s. GPS-based vehicle telematics has been around for 20 years. IoT is also not a separate market; it blends into current products and processes. Although much of the media reports on it as if it is a different animal (perhaps even the author of this book - guilty as charged?), you should not think of it this way.

The concept of constrained

The term constrained is an important concept in understanding IoT devices, data, and impacts on analytics. It refers to the limited battery power, bandwidth, and hardware capability that has to be considered in the design of IoT devices. For many IoT use cases, one or more of these has to be balanced with the need to record useful data.

IoT analytics challenges

There are some special challenges that come along with IoT data. The data was created by devices operating remotely, sometimes in widely varying environmental conditions that can change from day to day. The devices are often distributed widely geographically.

The data is communicated over long distances, often across different networking technologies. It is very common for data to first transmit across a wireless network, then through a type of gateway device to be sent over the public internet–which itself includes multiple different types of networking technology working together.

The data volume

A company can easily have thousands to millions of IoT devices with several sensors on each unit, each sensor reporting values on a regular basis. The inflow of data can grow quite large very quickly. Since IoT devices send data on an ongoing basis, the volume of data in total can increase much faster than many companies are used to.

To demonstrate how this can happen, imagine a company that manufactures small monitoring devices. It produces 12,000 devices a year, starting in 2010 when the product was launched. Each one is tested at the end of assembly and the values reported by the sensors on the device are kept for analysis for five years. The data growth looks like the following image:

A chart showing data storage needs for production snapshot of 200 KB and 1,000 units per month. Five years of production data is kept

Now, imagine the device also had internet connectivity to track sensor values, and each one remains connected for two years. Since the data inflow continues well after the devices are built, data growth is exponential until it stabilizes when older devices stop reporting values. This looks more like the blue area in the following chart:

Chart shows the addition of IoT data at 0.5 KB per message, 10 messages per day. Devices are connected for two years from production

In order to illustrate how large this can get, consider the following example. If you capture 10 messages per day and the message size is half of a full production snapshot, by 2017, data storage requirements would be over 1,500 times higher than production-only data.

For many companies, this introduces some problems. The database software, storage infrastructure, and available computing horsepower is not typically intended to handle this kind of growth. The licensing agreements with software vendors tends to be tied to the number of servers and CPU cores. Storage is handled by standard backup planning and retention policies.

The data volume rapidly leads to computing and storage requirements well beyond what can be held by a single server. It gets cost prohibitive very quickly under traditional architectures to distribute it across hundreds or thousands of servers. To do the best analytics, you need lots of historical data, and since you are unlikely to know ahead of time which data is most predictive, you have to keep as much as you can on hand.

With large-scale data, computing horsepower requirements for analytics are not very predictable and change dramatically depending on the question being asked. Analytic needs are very elastic. Traditional server planning ratchets up on premise resources with the anticipated number of servers needed to meet peak needs determined in advance. Doubling compute power in a short amount of time, if even possible, is very expensive.

IoT data volumes and computing resource requirements can quickly outpace all the other company data needs combined.

Problems with time

The only reason for time is so that everything doesn't happen at once.

– Albert Einstein

Time is very tightly tied to geographical position and the date on the calendar. The international standard way of tracking a common time is using Coordinated Universal Time (UTC). UTC is geographically tied to 00 longitude, which passes through Greenwich, England, in the UK. Although it is tied to the location, it is actually not the same as Greenwich Mean Time (GMT). GMT is a time zone, while UTC is a time standard. UTC does not observe Daylight Savings Time (DST):

Standard time zones of the World. Source: CIA Factbook

When data used for analytics is recorded at headquarters or a manufacturing plant, everything happens at the same place and time zone. IoT devices are spread out across the globe. Events that happen at the absolute same time do not happen at the same local time. How time is recorded affects the integrity of the resulting analytics.

When IoT devices communicate sensor data, time may be captured using the local time. It can dramatically affect analytics results if it is not clear whether local time or UTC was recorded. For example, consider an analyst working at a company that makes parking spot occupancy detection sensors. She is tasked with creating predictive models to estimate future parking lot fill rates. The time of day is likely to be a very predictive data point. It makes a big difference to her on how this time is recorded. Even determining if it is night or day at the sensor location will be difficult.

This may not be apparent to the engineer creating the device. His task is to design a device that determines if the spot is open or not. He may not appreciate the importance of writing code that captures a time value that can be aggregated across multiple time zones and locations.

There can also be issues with clock synchronization. Devices set their internal clock to be in sync with the time standard being used. If it is local time, it could be using the wrong time zone due to a configuration error. It could also get out of sync due to a communication problem with the time standard source.

If local time is being used, daylight savings time can cause problems. How will the events that happen between 1 a.m. and 2 a.m. on the day autumn daylight savings is adjusted be recorded since that hour happens twice? Laws that determine which days mark daylight savings time can change, as they did in Turkey when DST was scrapped in September 2016. If the device is locked into a set date range at the time of manufacture, the time would be incorrect for several days out of the year after the DST dates change.

How daylight savings time changes is different from country to country. In the United States, daylight savings time is changed at 02:00 local time in each time zone. In the European Union, it is coordinated so that all EU countries change at 01:00 GMT for all time zones at once. This keeps time zones always an hour apart at the expense of it changing at different local times for each time zone.

In early 2008, Central Brazil was one, two, or three hours ahead of eastern U.S., depending on the date

Source: Wikipedia commons

When time is recorded for an event, such as a parking spot being vacated, it is essential for analytics that the time is as close to the actual occurrence as possible. In practice, though, the time available for analytics can be the time the event occurred, the time the IoT device sent the data, the time the data was received, or the time the data was added to your data warehouse.

Problems with space

IoT devices are located in multiple geographic locations. Different areas of the world have different environmental conditions. Temperature variations can affect sensor accuracy. You could have less accurate readings in Calgary, Canada than in Cancun, Mexico, if cold impacts your device.

Elevation can affect equipment such as diesel engines. If location and elevation is not taken into consideration, you may falsely conclude from IoT sensor readings that a Denver-based fleet of delivery trucks is poorly managing fuel economy compared to a fleet in Indiana. Lots of mountain roads can burn up some fuel!

US elevation profile from LA to NYC. Source:

Remote locations may have weaker network access. The higher data loss could cause data values for those locations to be underrepresented in the resulting analytics.

Many IoT devices are solar powered. The available battery charge can affect the frequency of data reporting. A device in Portland, Oregon, where it is often cloudy and rainy will be more impacted than the same device in Phoenix, Arizona, where it is mostly sunny.

There are also political considerations related to the location of the IoT device. Privacy laws in Europe affect how the data from devices can be stored and what type of analytics is acceptable. You may be required to anonymize the data from certain countries, which can affect what you can do with analytics.

Data quality

Constrained devices means lossy networks. For analytics, it often results in either missing or inconsistent data. The missing data is often not random. As mentioned previously, it can be impacted by the location. Devices run on a software, called firmware, which may not be consistent across locations. This could mean differences in reporting frequency or formatting of values. It can result in lost or mangled data.

Data messages from IoT devices often require the destination to know how to interpret the message being sent. Software bugs can lead to garbled messages and data records.

Messages lost in translation or never sent due to dead batteries result in missing values. The conservation of power often means not all values available on the device are sent at the same time. The resulting datasets often have missing values, as the device sends some values consistently every time it reports and sends some other values less frequently.

Analytics challenges

Analytics often requires deciding on whether to fill in or ignore the missing values. Either choice may lead to a dataset that is not a representative of reality.

As an example of how this can affect results, consider the case of inaccurate political poll results in recent years. Many experts believe it is now in near crisis due to the shift of much of the world to mobile numbers as their only phone number. For pollsters, it is cheaper and easier to reach people on landline numbers. This can lead to the over representation of people with landlines. These people tend to be both older and wealthier than mobile-only respondents.

The response rate has also dropped from near 80% in the 1970s to about 8% (if you are lucky) today. This makes it more difficult (and expensive) to obtain a representative sample leading to many embarrassingly wrong poll predictions.

There can also be outside influences, such as environment conditions, that are not captured in the data. Winter storms can lead to power failures affecting devices that are able to report back data. You may end up drawing conclusions based on a non-representative sample of data without realizing it. This can affect the results of IoT analytics – and it will not be clear why.

Since connectivity is a new thing for many devices, there is also often a lack of historical data to base predictive models on. This can limit the type of analytics that can be done with the data.

It can also lead to a recency bias in datasets, as newer products are over represented in the data simply because a higher percentage are now a part of the IoT.

This leads us to the author's number one rule in IoT analytics:

Never trust data you don't know.

Treat it like a stranger offering you candy.

Business value concerns

Many companies are struggling to find value with IoT data. The costs to store, process, and analyze IoT data can grow quickly. With future financial returns uncertain, some companies are questioning if it is worth the investment.

According to McKinsey & Company, a consulting agency, most IoT data is not used. From their research, less than 1% of data generated by an oil platform was used for decision-making purposes.

Finding value with IoT analytics is often like finding a diamond in a mountain of rubble. We can accept that 1% of the data has value, but which 1% is it? This can vary depending on the question. One man's worthless granite is another man's priceless diamond.

The business value challenge is how to keep costs low while increasing the ability to create superior financial returns. Analytics is a great way to get there.


In this chapter, we defined, for the purposes of this book, what constitutes the IoT. We also defined what is meant by the term analytics when we use it here. We discussed special challenges that come with IoT data from volume of data to issues with time and space that are not normally a concern with internal company datasets. You should have a good idea of the scope of the book and the challenges that you will learn to overcome in the later chapters.

Left arrow icon Right arrow icon
Download code icon Download Code

Key benefits

  • •Make better business decisions and acquire greater control of your IoT infrastructure
  • •Learn techniques to solve unique problems associated with IoT and examine and analyze data from your IoT devices
  • •Uncover the business potential generated by data from IoT devices and bring down business costs


We start with the perplexing task of extracting value from huge amounts of barely intelligible data. The data takes a convoluted route just to be on the servers for analysis, but insights can emerge through visualization and statistical modeling techniques. You will learn to extract value from IoT big data using multiple analytic techniques. Next we review how IoT devices generate data and how the information travels over networks. You’ll get to know strategies to collect and store the data to optimize the potential for analytics, and strategies to handle data quality concerns. Cloud resources are a great match for IoT analytics, so Amazon Web Services, Microsoft Azure, and PTC ThingWorx are reviewed in detail next. Geospatial analytics is then introduced as a way to leverage location information. Combining IoT data with environmental data is also discussed as a way to enhance predictive capability. We’ll also review the economics of IoT analytics and you’ll discover ways to optimize business value. By the end of the book, you’ll know how to handle scale for both data storage and analytics, how Apache Spark can be leveraged to handle scalability, and how R and Python can be used for analytic modeling.

What you will learn

•Overcome the challenges IoT data brings to analytics •Understand the variety of transmission protocols for IoT along with their strengths and weaknesses •Learn how data flows from the IoT device to the final data set •Develop techniques to wring value from IoT data •Apply geospatial analytics to IoT data •Use machine learning as a predictive method on IoT data •Implement best strategies to get the most from IoT analytics •Master the economics of IoT analytics in order to optimize business value

Product Details

Country selected

Publication date : Jul 24, 2017
Length 378 pages
Edition : 1st Edition
Language : English
ISBN-13 : 9781787120730
Category :

What do you get with eBook?

Product feature icon Instant access to your Digital eBook purchase
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
Buy Now

Product Details

Publication date : Jul 24, 2017
Length 378 pages
Edition : 1st Edition
Language : English
ISBN-13 : 9781787120730
Category :

Table of Contents

20 Chapters
Title Page Chevron down icon Chevron up icon
Credits Chevron down icon Chevron up icon
About the Author Chevron down icon Chevron up icon
About the Reviewer Chevron down icon Chevron up icon Chevron down icon Chevron up icon
Customer Feedback Chevron down icon Chevron up icon
Preface Chevron down icon Chevron up icon
1. Defining IoT Analytics and Challenges Chevron down icon Chevron up icon
2. IoT Devices and Networking Protocols Chevron down icon Chevron up icon
3. IoT Analytics for the Cloud Chevron down icon Chevron up icon
4. Creating an AWS Cloud Analytics Environment Chevron down icon Chevron up icon
5. Collecting All That Data - Strategies and Techniques Chevron down icon Chevron up icon
6. Getting to Know Your Data - Exploring IoT Data Chevron down icon Chevron up icon
7. Decorating Your Data - Adding External Datasets to Innovate Chevron down icon Chevron up icon
8. Communicating with Others - Visualization and Dashboarding Chevron down icon Chevron up icon
9. Applying Geospatial Analytics to IoT Data Chevron down icon Chevron up icon
10. Data Science for IoT Analytics Chevron down icon Chevron up icon
11. Strategies to Organize Data for Analytics Chevron down icon Chevron up icon
12. The Economics of IoT Analytics Chevron down icon Chevron up icon
13. Bringing It All Together Chevron down icon Chevron up icon

Customer reviews

Top Reviews
Rating distribution
Empty star icon Empty star icon Empty star icon Empty star icon Empty star icon 0
(0 Ratings)
5 star 0%
4 star 0%
3 star 0%
2 star 0%
1 star 0%
Top Reviews
No reviews found
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial


How do I buy and download an eBook? Chevron down icon Chevron up icon

Where there is an eBook version of a title available, you can buy it from the book details for that title. Add either the standalone eBook or the eBook and print book bundle to your shopping cart. Your eBook will show in your cart as a product on its own. After completing checkout and payment in the normal way, you will receive your receipt on the screen containing a link to a personalised PDF download file. This link will remain active for 30 days. You can download backup copies of the file by logging in to your account at any time.

If you already have Adobe reader installed, then clicking on the link will download and open the PDF file directly. If you don't, then save the PDF file on your machine and download the Reader to view it.

Please Note: Packt eBooks are non-returnable and non-refundable.

Packt eBook and Licensing When you buy an eBook from Packt Publishing, completing your purchase means you accept the terms of our licence agreement. Please read the full text of the agreement. In it we have tried to balance the need for the ebook to be usable for you the reader with our needs to protect the rights of us as Publishers and of our authors. In summary, the agreement says:

  • You may make copies of your eBook for your own use onto any machine
  • You may not pass copies of the eBook on to anyone else
How can I make a purchase on your website? Chevron down icon Chevron up icon

If you want to purchase a video course, eBook or Bundle (Print+eBook) please follow below steps:

  1. Register on our website using your email address and the password.
  2. Search for the title by name or ISBN using the search option.
  3. Select the title you want to purchase.
  4. Choose the format you wish to purchase the title in; if you order the Print Book, you get a free eBook copy of the same title. 
  5. Proceed with the checkout process (payment to be made using Credit Card, Debit Cart, or PayPal)
Where can I access support around an eBook? Chevron down icon Chevron up icon
  • If you experience a problem with using or installing Adobe Reader, the contact Adobe directly.
  • To view the errata for the book, see and view the pages for the title you have.
  • To view your account details or to download a new copy of the book go to
  • To contact us directly if a problem is not resolved, use
What eBook formats do Packt support? Chevron down icon Chevron up icon

Our eBooks are currently available in a variety of formats such as PDF and ePubs. In the future, this may well change with trends and development in technology, but please note that our PDFs are not Adobe eBook Reader format, which has greater restrictions on security.

You will need to use Adobe Reader v9 or later in order to read Packt's PDF eBooks.

What are the benefits of eBooks? Chevron down icon Chevron up icon
  • You can get the information you need immediately
  • You can easily take them with you on a laptop
  • You can download them an unlimited number of times
  • You can print them out
  • They are copy-paste enabled
  • They are searchable
  • There is no password protection
  • They are lower price than print
  • They save resources and space
What is an eBook? Chevron down icon Chevron up icon

Packt eBooks are a complete electronic version of the print edition, available in PDF and ePub formats. Every piece of content down to the page numbering is the same. Because we save the costs of printing and shipping the book to you, we are able to offer eBooks at a lower cost than print editions.

When you have purchased an eBook, simply login to your account and click on the link in Your Download Area. We recommend you saving the file to your hard drive before opening it.

For optimal viewing of our eBooks, we recommend you download and install the free Adobe Reader version 9.