Introducing NLP concepts – tokens and lemmas

Let’s begin exploring Advanced Features by creating a simple summarization function in Python and Streamlit. Although many packages and libraries offer powerful summarization capabilities, this book focuses on web application development rather than NLP or summarization.

Adding the summarization function

Though the name is self-explanatory, a summarization function is a piece of code that summarizes a sentence or a text, extracting only the most important part of it. This task can be achieved in many ways – some very easy, like the one we are proposing just to show how to develop complex web applications with Streamlit, and some very sophisticated, leveraging artificial intelligence and neural networks. Figure 5.21 shows the code where we add the summarize_text function:

Figure 5.21: The summarize_text function

An attention point is that after the NLP packages, we imported two new...