Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds
Arrow up icon
GO TO TOP
Python Data Science Essentials

You're reading from   Python Data Science Essentials Learn the fundamentals of Data Science with Python

Arrow left icon
Product type Paperback
Published in Oct 2016
Publisher Packt
ISBN-13 9781786462138
Length 378 pages
Edition 2nd Edition
Languages
Arrow right icon
Authors (2):
Arrow left icon
Alberto Boschetti Alberto Boschetti
Author Profile Icon Alberto Boschetti
Alberto Boschetti
Luca Massaron Luca Massaron
Author Profile Icon Luca Massaron
Luca Massaron
Arrow right icon
View More author details
Toc

A peek at Natural Language Processing (NLP)


This section is not strictly related to machine learning, but it contains some machine learning results in the area of Natural Language Processing. Python has many toolkits to process text data, but the most powerful and complete toolkit is NLTK, the Natural Language Tool Kit.

In the following sections, we'll explore its core functionalities. We will work on the English language; for other languages, you will first need to download the language corpora (note that sometimes languages have no free open source corpora for NLTK).

Refer to the official website of NLTK data, http://www.nltk.org/nltk_data/, to have access to corporas and lexical resources in many languages, ready to work with NLTK.

Word tokenization

Tokenization is the action of splitting text into words. Chunking whitespace seems very easy, but it's not, because text contains punctuation and contractions. Let's start with an example:

In: my_text = "The coolest job in the next 10 years will...
lock icon The rest of the chapter is locked
CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Python Data Science Essentials
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at $19.99/month. Cancel anytime
Modal Close icon
Modal Close icon