You're reading from Hands-On Machine Learning with Azure

Product typeBook

Published inOct 2018

PublisherPackt

ISBN-139781789131956

Edition1st Edition

Tools

Azure

Concepts

Machine Learning

Authors (5):

Thomas K Abraham

Parashar Shah

Jen Stirrup

Lauri Lehman

Anindita Basak

View More author details

HDInsight

HDInsight is a type of implementation of Hadoop that runs on the Microsoft Azure platform. HDInsight builds on the Hortonworks Data Platform (HDP), and is completely compatible with Apache Hadoop.

HDInsight can be perceived as Microsoft's Hadoop-as-a-Service (Haas). You can quickly deploy the system from a portal or through Windows PowerShell scripting, without having to create any physical or virtual machines.

The following are features of HDInsights:

You can implement a small or large number of nodes in a cluster
You pay only for what you use
When your job is complete, you can deprovision the cluster and, of course, stop paying for it
You can use Microsoft Azure Storage so that even when the cluster is deprovisioned, you can retain the data
The HDInsight service works with input-output technologies from Microsoft and other vendors

As mentioned, the HDInsight...

R with HDInsight

What are the main features of HDInsight? It is a Microsoft proprietary solution, but it is a 100% Apache Hadoop solution in the Microsoft Azure cloud. Azure HDInsight is a service that deploys and provisions Apache Hadoop clusters in the cloud for big data analytics.

HDInsight provides a software framework designed to manage, analyze, and report on big data. You can use HDInsight to perform interactive queries at petabyte scales over structured or unstructured data in any format. You can also build models, connecting them to BI tools. HDInsight is aimed at providing big data analytics and insights through Excel and Power BI. Azure's HDInsight service makes Apache Hadoop available as a service in the cloud, providing a software framework designed to manage, analyze, and report on big data. As a cloud-based service, it makes these resources available in a simpler...

Getting started with Azure HDInsight and ML services

HDInsight has a number of cluster types, which include Hadoop (Hive), HBase, Storm, Spark, Kafka, Interactive Hive (LLAP), and ML Services (R Server) (with R Studio, R 9.1). Here is the ML Cluster configuration, which is established during setup:

Setup and configuration of HDInsight

In this section, we will set up and configure HDInsight. To set up and configure HDInsight, carry out the following steps:

Ensure that you have an Azure account
Log into the Azure portal at portal.azure.com
When you are logged into the Azure portal, click on the button to add a new resource
In the search query box, type in HDInsight and you will be given a number of options
Select the option...

HDInsight and data analytics with R

Firstly, we need to get our data into Azure so that HDInsight can see it. We can upload data directly to Azure Storage, or we can use functionality in SQL Server Integration Services (SSIS). SSIS has the capability of connecting to Azure Blob Storage and Azure HDInsight. It enables you to create integration service packages that transfer data between an Azure Blob Storage and the on-premise data source. Then, the Azure HDInsight process can conduct processing on the data.

In order to get the data into HDInsight using SSIS, it's necessary to install the Azure Feature Pack. The Microsoft SSIS Feature Pack for Azure provides SQL Server Integration Services with the capability to connect to many Azure services, such as Azure Blob Storage, Azure Data Lake Store, Azure SQL Data Warehouse, and Azure HDInsight. It is a separate install, and you...

Enriching data for analysis

With big data solutions, sometimes the data needs to be transformed and processed into smaller chunks due to its sheer size. In order to deal with this problem, Microsoft have introduced some functionality to help. This section will cover the features that are designed to assist with big data issues.

rxDataSteps

The rxDataStep function can be used to process data in chunks. It is one of the important data transformation functions in Microsoft ML Services.

The rxDataStep function can be used to create and transform subsets of data. The rxDataStep function processes data one chunk at a time, reading from one data source and writing to another. rxDataStep allows you to modify existing columns or add...

Summary

In this chapter, we have examined the machine learning process for Microsoft ML Services on HDInsight. We have reviewed how to ingest data, how to clean it, how to model it, and how to visualize it.

The final step is to ensure that HDInsight is torn down when you have stopped using it. HDInsight is charged by the minute and it will cost you money to leave it running. It is recommended that you save code and tear everything down when you no longer need it.

If you are simply wanting to run code and learn how to use Microsoft ML Services, the previous code samples will also work on Microsoft ML Server in Visual Studio on the DSVM.

The rest of the chapter is locked

You have been reading a chapter from

Hands-On Machine Learning with Azure

Published in: Oct 2018Publisher: PacktISBN-13: 9781789131956

A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.

undefined

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Start free trial

Renews at $15.99/month. Cancel anytime

Authors (5)

Thomas K Abraham

Dr. Thomas K Abraham is a cloud solution architect (advanced analytics and AI) at Microsoft in the South Central Region of the USA. Since January 2016, he's been assisting organizations in leveraging technologies such as SQL, Spark, Hadoop, NoSQL, BI, and AI on Azure. Prior to that, Thomas spent 10 years in Ecolab, where he designed algorithms for IoT devices and built solutions for anomaly detection. In the oil and gas division, he designed and built customer-facing analytics solutions for multiple super majors. His work was focused on preventing equipment failure by modeling corrosion, scale, and other stresses. He has a PhD in Chemical Engineering from The Ohio State University in 2005. His thesis focused on the use of nonlinear optimization with reaction models.
Read more about Thomas K Abraham

Parashar Shah

Parashar Shah is a Senior Program Manager in the Azure Machine Learning platform team.Currently, he works on making Azure Machine Learning services the best place to do e2e machine learning for building custom AI solutions using big data. Previously at Microsoft, he has been a Data Scientist and a Data Solutions Architect in various Cloud and AI teams. Prior to joining Microsoft, Parashar worked at Nokia Networks as a Solutions Architect & Product Manager building customer experience analytics solutions for global telcos. He also co-founded a carpooling startup, which helped employees carpool safely. He has 10+ years of global work experience. He is an alum of Indian Institute of Management, Bangalore and Gujarat University.
Read more about Parashar Shah

Jen Stirrup

Jen Stirrup is a data strategist and technologist, a Microsoft Most Valuable Professional (MVP), and a Microsoft Regional Director, a tech community advocate, a public speaker and blogger, a published author, and a keynote speaker. Jen is the founder of a boutique consultancy based in the UK, Data Relish, which focuses on delivering successful business intelligence and artificial intelligence solutions that add real value to customers worldwide. She has featured on the BBC as a guest expert on topics relating to data.
Read more about Jen Stirrup

Lauri Lehman

Lauri Lehman is a data scientist who is focused on machine learning tools in Azure. He helps customers to design and implement machine learning solutions in the cloud. He works for the software consultancy company, Zure, based in Helsinki, Finland. For the past 4 years, Lauri has specialized in data and machine learning in Azure. He has worked on many machine learning projects, developing solutions for demand estimation, text analytics, and image recognition, for example. Lauri has previously worked as an academic researcher in theoretical physics, after obtaining his PhD on topological quantum walks. He still likes to follow the progress of modern physics and is eagerly a waiting the era of quantum machine learning!
Read more about Lauri Lehman

Anindita Basak

Anindita Basak is a cloud architect with almost 15+ years of experience, the last 12 years of which she has been extensively working on Azure. She has delivered various real-time implementations on Azure data analytics, and cloud-native and real-time event-driven architecture for Fortune 500 enterprises, ranging from banking, financial services, and insurance (BFSI)to retail sectors. She is also a cloud and DataOps trainer and consultant, and author of cloud AI and DevOps books.
Read more about Anindita Basak

Other recommended products

Related to this chapter

Learning Microsoft Cognitive Services

This book will introduce you to the APIs released as part of Cognitive Services platform and show you how to leverage their capabilities. More importantly, you'll see how the power of these APIs can be combined to build real-world apps that have cognitive capabilities.

BookSep 2018312 pages

Hands-On Data Warehousing with Azure Data Factory

Azure Data Factory (ADF) is a Microsoft Azure PaaS solution which supports data movement between many on premises and cloud data sources. This book covers custom tailored tutorials to help you develop , maintain and troubleshoot data movement processes and environments using Azure Data Factory V2 and SQL Server Integration Services 2017

BookMay 2018284 pages

Deep Learning with Microsoft Cognitive Toolkit Quick Start Guide

Cognitive Toolkit is one of the most popular and recently open sourced deep learning toolkit by Microsoft. Cognitive Toolkit is used to train fast and effective deep learning models. This book will be a quick introduction to using Cognitive Toolkit and will teach you how to train and validate different types of neural networks.

BookMar 2019208 pages

SQL Server 2017 Machine Learning Services with R

With integrated R Services within SQL Server 2017, developers and data scientists can now benefit from the integrated, effective, efficient and more streamlined analytics environment. In this book, you will understand how to leverage the capabilities of R Services in SQL Server 2017. This short yet effective guide will help you get familiar with SQL Server 2017 R Services, and will show how to implement efficient data science models using it.

BookFeb 2018338 pages

Advanced Analytics with R and Tableau

R is the go-to tool for statistics and data mining while Tableau offers an interface to filter data, plug and play with rich visualizations to describe insights from your data. When combined these two tools makes it easier to harness interesting patterns and communicate stories. This book covers various analytical techniques like prediction, classification, clustering and best practices to visualize it using interactive dashboard with drop-downs, sliders, and other visual cues of Tableau. Get to know how R can be used in conjunction with Tableau and implement powerful machine learning techniques making big data analytics accessible and presentable through Tableau workbooks.

BookAug 2017178 pages

Azure Data Factory Cookbook

With the help of well-structured and practical recipes, this book will teach you how to integrate data from the cloud and on-premise. You’ll learn how to transform, clean, and consolidate data into a single data platform and get to grips with using ADF as the main ETL and orchestration tool for your data warehouse or data platform project.

BookDec 2020382 pages

Automated Machine Learning

This guide will help you to explore automated machine learning (AutoML), a rapidly growing subfield of machine learning. You’ll learn how you can use AutoML to fully automate the machine learning process even if you’re not an expert, and in turn increase your productivity drastically.

BookFeb 2021312 pages

Cloud Analytics with Microsoft Azure

Cloud Analytics with Microsoft Azure is an end-to-end guide to processing and analyzing big data using a range of Microsoft Azure features. This book covers everything you need to build your own data warehouse and learn numerous techniques to gain useful insights by analyzing big data.

BookNov 2019242 pages

Stream Analytics with Microsoft Azure

This book is your guide to understanding the basics of how Azure Stream Analytics works, and build your own analytics solution using its capabilities. By the end of this book, you will be well-versed in using Azure Stream Analytics to develop an efficient analytics solution which can work with any type of data.

BookDec 2017322 pages

Limitless Analytics with Azure Synapse

This book helps you understand the basic concepts and techniques of using Azure Synapse step-by-step. You'll gradually gain the skills you need to work with data and develop analytics solutions using the Azure analytics platform even with no prior knowledge of Azure.

BookJun 2021392 pages

Architecting Microsoft Azure Solutions - Exam Guide 70-535

This study guide includes all the topics that are still relevant from the previous 70-534 exam, updated with the latest features like Artificial Intelligence, IoT, and architecture styles. This guide will help Azure Architects, Developers or anyone interested in designing and implementing effective Cloud architecture strategies.

BookApr 2018418 pages

Serverless Integration Design Patterns with Azure

Whether you work for a small start-up or for a large enterprise, this book can help you understand Microsoft Cloud Integration technologies to Integrate application and business processes. By using this book, readers will be able to learn various Architecture design principles while connecting Enterprise application with Azure components.

BookFeb 2019494 pages

Personalised recommendations for you

Based on your interests and search pattern

Et al.

Ever wonder why speech recognition systems don't understand the Scottish accent, or what would happen if an astronaut only ate mac 'n' cheese, or other spurious reflections you'd have at a bar? We did, then collated those deliberations into absurd research articles with fake figures and methodologies inspired by even more fictionally absurd studies.

BookAug 2023230 pages5

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages4

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages5

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages1

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages5

Mastering Tableau 2023

This book is a comprehensive resource to mastering your Tableau skills and becoming a BI expert. As you progress, you will learn how to build advanced dashboards and improve your storytelling to derive key business insight, as well as make you well-versed with advanced functionalities of Tableau in the business intelligence domain.

BookAug 2023684 pages

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

BookSep 2023258 pages5

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

BookSep 2023258 pages2

Data Engineering with AWS

Embark on a journey to master data engineering pipelines on AWS! Our book offers a hands-on experience of AWS services for ingesting, transforming, and consuming data. Whether you're an absolute beginner or someone with basic data engineering experience, this guide is an indispensable resource.

BookOct 2023636 pages5

Modern Data Architecture on AWS

Every organization wants an agile, performant, and cost-effective data platform that meets all their current and future business needs. Purpose-built AWS analytics services and their features play a big part in building such a modern data platform. This book brings to you all the design and architectural patterns that’ll help you achieve this goal.

BookAug 2023420 pages5

Practical Guide to Applied Conformal Prediction in Python

Discover the power of Conformal Prediction with the "Practical Guide to Applied Conformal Prediction in Python." Master the latest techniques to quantify uncertainty in machine learning and computer vision models, and seamlessly apply them to your industry applications.

BookDec 2023240 pages

TinyML Cookbook

With over 70 project-based recipes, the TinyML Cookbook is a practical guide that will help you to get the most out of your microcontrollers. It provides a comprehensive understanding of the theoretical foundations while giving you hands-on experience training ML models for deployment on Arduino Nano 33 BLE Sense, Raspberry Pi Pico, and SparkFun RedBoard Artemis Nano microcontrollers.

BookNov 2023664 pages