Reader small image

You're reading from  R Machine Learning Projects

Product typeBook
Published inJan 2019
Reading LevelExpert
PublisherPackt
ISBN-139781789807943
Edition1st Edition
Languages
Right arrow
Author (1)
Dr. Sunil Kumar Chinnamgari
Dr. Sunil Kumar Chinnamgari
author image
Dr. Sunil Kumar Chinnamgari

Dr. Sunil Kumar Chinnamgari has a Ph.D. in computer science and specializes in machine learning and natural language processing. He is an AI researcher with more than 14 years of industry experience. Currently, he works in the capacity of lead data scientist with a US financial giant. He has published several research papers in Scopus and IEEE journals and is a frequent speaker at various meetups. He is an avid coder and has won multiple hackathons. In his spare time, Sunil likes to teach, travel, and spend time with family.
Read more about Dr. Sunil Kumar Chinnamgari

Right arrow

Understanding language models

In the English language, the character a appears much more often in words and sentences than the character x. Similarly, we can also observe that the word is occurs more frequently than the word specimen. It is possible to learn the probability distributions of characters and words by examining large volumes of text. The following screenshot is a chart showing the probability distribution of letters given a corpus (text dataset):

Probability distribution of letters in a corpus

We can observe that the probability distributions of characters are non-uniform. This essentially means that we can recover the characters in a word, even if they are lost due to noise. If a particular character is missing in a word, it can be reconstructed just based on the characters that are surrounding the missing character. The reconstruction of the missing character is...

lock icon
The rest of the page is locked
Previous PageNext Page
You have been reading a chapter from
R Machine Learning Projects
Published in: Jan 2019Publisher: PacktISBN-13: 9781789807943

Author (1)

author image
Dr. Sunil Kumar Chinnamgari

Dr. Sunil Kumar Chinnamgari has a Ph.D. in computer science and specializes in machine learning and natural language processing. He is an AI researcher with more than 14 years of industry experience. Currently, he works in the capacity of lead data scientist with a US financial giant. He has published several research papers in Scopus and IEEE journals and is a frequent speaker at various meetups. He is an avid coder and has won multiple hackathons. In his spare time, Sunil likes to teach, travel, and spend time with family.
Read more about Dr. Sunil Kumar Chinnamgari