Reader small image

You're reading from  Distributed Data Systems with Azure Databricks

Product typeBook
Published inMay 2021
Reading LevelBeginner
PublisherPackt
ISBN-139781838647216
Edition1st Edition
Languages
Concepts
Right arrow
Author (1)
Alan Bernardo Palacio
Alan Bernardo Palacio
author image
Alan Bernardo Palacio

Alan Bernardo Palacio is a data scientist and an engineer with vast experience in different engineering fields. His focus has been the development and application of state-of-the-art data products and algorithms in several industries. He has worked for companies such as Ernst and Young, Globant, and now holds a data engineer position at Ebiquity Media helping the company to create a scalable data pipeline. Alan graduated with a Mechanical Engineering degree from the National University of Tucuman in 2015, participated as the founder in startups, and later on earned a Master's degree from the faculty of Mathematics in the Autonomous University of Barcelona in 2017. Originally from Argentina, he now works and resides in the Netherlands.
Read more about Alan Bernardo Palacio

Right arrow

Reading a dataset

Reading datasets using Petastorm can be very simple. In this section, we will demonstrate how we can easily load a Petastorm dataset into two frequently used deep learning frameworks, which are TensorFlow and PyTorch:

  1. To load our Petastorm datasets, we use the petastorm.reader.Reader class, which implements the iterator interface that allows us to use plain Python to go over the samples very efficiently. The petastorm.reader.Reader class can be created using the petastorm.make_reader factory method:
    from petastorm import make_reader
    with make_reader('dfs://some_dataset') as reader:
       for sample in reader:
           print(sample.id)
           plt.imshow(sample.image1)
  2. The following code example shows how we can stream a dataset into the TensorFlow Examples class, which as we have seen before is a named tuple with the keys being the ones specified in the Unischema of...
lock icon
The rest of the page is locked
Previous PageNext Page
You have been reading a chapter from
Distributed Data Systems with Azure Databricks
Published in: May 2021Publisher: PacktISBN-13: 9781838647216

Author (1)

author image
Alan Bernardo Palacio

Alan Bernardo Palacio is a data scientist and an engineer with vast experience in different engineering fields. His focus has been the development and application of state-of-the-art data products and algorithms in several industries. He has worked for companies such as Ernst and Young, Globant, and now holds a data engineer position at Ebiquity Media helping the company to create a scalable data pipeline. Alan graduated with a Mechanical Engineering degree from the National University of Tucuman in 2015, participated as the founder in startups, and later on earned a Master's degree from the faculty of Mathematics in the Autonomous University of Barcelona in 2017. Originally from Argentina, he now works and resides in the Netherlands.
Read more about Alan Bernardo Palacio