Chapter 10. Closing Thoughts on Big Data
We have covered a broad range of topics thus far. We have looked at technologies used for big data, for data science, and for machine learning. We have learned about how companies are implementing their big data corporate strategies. We have also developed a handful of real-world applications along the way.
This chapter discusses the practical considerations of big data or data science initiatives at corporations. The field is continually evolving, with the introduction of newer technologies, newer open source tools, and new concepts in data mining. Due to this, organizations of all sizes share common challenges.
Data science success stories are everywhere in the media. In fact, most, if not all, of the investment happening in technology today has some connection to aspects of data science. Indeed, it has become an indispensable and integral aspect of IT development.
In this chapter, we will discuss a few of the common themes of implementing data science...
Corporate big data and data science strategy
You have read about it in the papers, you have seen it on the evening news, you have heard about it from your friends – big data and data science are everywhere and they are here to stay.
The success stories from Silicon Valley have made the effect even more pronounced. Who would have thought that a ride-sharing and ride-hailing phone application, Uber, could become one of the most popular companies in the world with an estimated valuation of close to $70 billion. Sites and apps such as Airbnb turned apartment-sharing into a booming business, becoming the second most valued company at $30 billion.
These and other similar events transformed the topics of big data and data science from being purely theoretical and technical subjects into common terminology that people have come to associate with unbounded investment success.
Since nearly all major technology vendors have started adding features categorized as big data, nearly all companies that invest...
Big data often involves the gathering of large volumes of data that may contain users' personal information. Companies such as Facebook and Google have flourished on analyzing individual information to target ads and perform other types of marketing. This evidently poses an ethical dilemma. To what extent should personal data be collected? And how much is too much? There are, of course, no correct answers to these questions. The rise of hacking in which information from hundreds of millions of user accounts has been compromised is so commonplace today that we have almost become complacent about the consequences.
In October 2017, Yahoo! disclosed that 3 billion accounts, in fact every single account on Yahoo!, had suffered a data breach. Equifax, one of the largest credit reporting companies in the US suffered a data breach that exposed the personal details of more than 140 million consumers. There were scores of other similar incidents, and in all of them, the common...
Silicon Valley and data science
Several of the key innovations we see today in big data have emerged from Silicon Valley. The region has been a tech hub for decades and has launched some of the most successful companies such as Apple, Google, Facebook, and eBay. The presence of universities, such as the University of California at Berkeley, has made access to talent relatively easy.
That said, the cost of living in the region has sky-rocketed, especially in the wake of the growth of the big data and data science industry. Today, the average rent for a one-bedroom apartment is well above $3,500 per month, making it more expensive than even New York City.
Silicon Valley, however, is synonymous with success and many new entrepreneurs are drawn to the region. Startups have sprung up, many of them commanding tens of millions of dollars in VC investment. However, entrepreneurs should heed statistical warnings given the high failure percentage of startup businesses. It is one thing to have a great...
The significant advantages of big data and data science notwithstanding, their successes and breakthrough growth, it is still important to bear in mind that the element of human thinking is essential in all endeavors.
Big data technologies will allow us to analyze data more efficiently. But we still need to use proper judgment to decide on our ideal use cases. This is not trivial. Large companies find initiatives just as challenging (although at a higher scale) as seasoned big data professionals.
Similarly, data science and machine learning can empower us to make predictions and gain foresight with the help of sophisticated algorithms and code. However, it is still incumbent upon the user to evaluate the results and make decisions not solely based on the predicted output. Users should apply common sense and experience in making such assessments. If the GPS instructs the driver to go on a certain road on a snowy winter night and the driver knows that the road won't have been...
Overall, while the path to big data and data science success may seem arduous, hopefully the preceding chapters have provided a comprehensive overview of various topics in big data. We discussed data mining and machine learning, learned about the various tools and technologies in the respective disciplines, and developed applications on real-world data and provided parting thoughts on the nuances of organizational big data and data science initiatives.
The next few pages list some links to resources that the reader may find useful for learning more about the respective subject areas.
