Toward Syntax-Free Semantic Role Labeling with ChatGPT and GPT-4

Transformers have made more progress in the past few years than NLP in the past generation. Former NLP models would be trained to understand a language’s basic syntax before running Semantic Role Labeling (SRL). The NLP software contained syntax trees, rule bases, and parsers. The performance of such systems was limited by the number of combinations of words that led to an infinity of contexts.

Shi and Lin (2019) started their paper by asking if preliminary syntactic and lexical training can be skipped. Could a system become “syntax-free” and understand language without relying on pre-designed syntax trees? Could a BERT-based model perform SRL without going through those classical training phases? The answer is yes!

Shi and Lin (2019) suggested that SRL can be considered sequence labeling and provide a standardized input format. Since then, OpenAI has reached near human-level syntax-free...

Getting started with cutting-edge SRL

Chapter 7, The Generative AI Revolution with ChatGPT, introduced the complexity of adapting to the widening taxonomy of deep learning models in general, and LLMs, such as ChatGPT, in particular.

SRL provides enhanced information extraction on each word’s role in a sentence. The extracted information can improve translation, summarization, and other NLP tasks. A model that understands the semantic role of words will provide better-quality NLP outputs.

However, choosing a path to implement an NLP task has become challenging. SRL is no exception. We can sum up the issues to examine with four parameters that describe the resources used in this chapter: task-specific, general-purpose, development, and self-service:

Figure 12.1: The complexity of LLM management

Figure 12.1 shows the complexity of LLM project management. The parameters interact in various possible ways for a project. The risk of each parameter must be carefully...

Entering the syntax-free world of AI

SRL is as difficult for humans as for machines. However, transformers have taken us to the disruptive boundary of our human baselines.

A syntax-free approach to SRL is quite innovative. Classical NLP methods include dependency analysis, Part-of-Speech (POS) parsing, and learning about phrase structure. The classical approach trains the model to understand a sentence’s grammatical structure and syntax.

The model designed by Shi and Lin (2019) doesn’t apply an explicit syntax analysis process. The BERT model relies on its unique ability to understand the structure of a sentence. It implicitly captures syntactic features from the vast amount of data it learns how to represent. OpenAI took this approach further and decided to let its GPT models learn syntax without going through the tedious process of training a model for specific tasks, including SRL. As such, there are no syntax trees in a GPT model (syntax-free).

In this...

Defining SRL

Shi and Lin (2019) advanced and proved that we can find who did what and where without depending on lexical or syntactic features. This chapter is based on Peng Shi and Jimmy Lin’s research at the University of Waterloo, California. They showed how transformers learn language structures better with attention layers.

SRL labels the semantic role as the role a word or group of words plays in a sentence and the relationship established with the predicate.

A semantic role is the role a noun or noun phrase plays in relation to the main verb in a sentence. For example, in the sentence Marvin walked in the park, Marvin is the agent of the event occurring in the sentence. The agent is the doer of the event. The main verb, or governing verb, is walked.

The predicate describes something about the subject or agent. The predicate could be anything that provides information on the features or actions of a subject. In our approach, we will refer to the predicate...

SRL experiments with ChatGPT with GPT-4

We will run our SRL experiments with ChatGPT with GPT-4, beginning with a basic sample and then challenging the model with a more complex example to explore the system’s capacity and limits. You can access ChatGPT on OpenAI’s platform: https://chat.openai.com/.

ChatGPT has two revolutionary features:

• It is syntax-free, meaning that it does not rely on syntax trees or rules at all. This approach is a paradigm shift from classical AI to generative models. Generative models detect statistical patterns in sequences but do not learn rules at all. The rules are implicit through statistical training, not explicit.
• The responses are not pre-designed and remain stochastic, meaning that we will get a mostly reliable (like for any AI model) output but not the same word-for-word output each time. This stochastic, random behavior makes recent LLMs so human-like.

Let’s begin with a basic sample.

Basic...

Questioning the scope of SRL

We are alone when faced with a real-life project. We have a job to do, and the only people to satisfy are those who asked for that project.

Pragmatism must come first – technical ideology after.

In the 2020s, former AI ideology and new ideology coexist. By the decade’s end, the two worlds will have merged to form a new era.

This section questions the productivity of SRL and its motivation through two aspects:

• The limit of predicate analysis
• Questioning the use of the term “semantic”

The challenges of predicate analysis

SRL relies on predicates. SRL only works as long as you provide a verb. But millions of sentences do not contain verbs (ellipsis).

If you provide an LLM with an assertion alone, it works.

But what happens if your assertion is an answer to a question? The question can contain a verb, for example:

Person 1: What would you like to drink, please?

But the person...

Redefining SRL

SRL presupposes that sentences contain predicates, which is often a false assumption. Analyzing a sentence cannot be based on a predicate analysis alone.

A predicate contains a verb. The predicate tells us more about the subject. The following predicate contains a verb and additional information:

The dog ate his food quickly.

ate...quickly tells us more about the way the dog ate. However, a verb alone can be a predicate, as in Dogs eat.

The problem here resides in the fact that “verbs” and “predicates” are part of syntax and grammar analysis, not semantics.

Understanding how words fit together from a grammatical, functional point of view is restrictive.

Take this sentence that means absolutely nothing:

Globydisshing maccaked up all the tie.

However, SRL could perform “semantic” analysis on a sentence that means nothing:

Figure 12.6: Analyzing a meaningless sentence

We can draw some...

From task-specific SRL to emergence with ChatGPT

We have seen that OpenAI’s ChatGPT with GPT-4 has taken LLMs further for various tasks, including SRL. Thus, general-purpose LLMs do not necessarily need to learn syntax explicitly. They don’t need to learn the rules and principles of syntax that explain how to form phrases, clauses, and sentences. They can explain sentences with and beyond SRL.

We have gone through the main aspects of SRL in this chapter with several examples. This section will focus on running GPT-4 through the API to explore its ability to perform SRL without explicitly being trained for this task.

Open Semantic_Role_Labeling_GPT-4.ipynb in the directory of this chapter in the GitHub repository.

We will first install and import OpenAI.

1. Installing OpenAI

The program updates pip and installs OpenAI:

!pip install --upgrade pip
#Importing openai
try:
  import openai
except:
  !pip install openai -qq
  import openai
from openai...

Summary

In this chapter, we explored SRL. SRL tasks are difficult for both humans and machines. Transformer models have shown that human baselines can be reached for many NLP topics to a certain extent.

We first defined the revolutionary syntax-free world of recent LLM models. AI is experiencing a significant paradigm shift from task-specific training to general-purpose Generative AI models such as OpenAI’s GPT series.

We ran several examples with a general-purpose model using ChatGPT with GPT-4. We confirmed that there is no silver bullet and that the ultimate choice will depend on the goals of a project.

We found that a general-purpose LLM model can perform predicate sense disambiguation. We ran basic examples in which a transformer could identify the meaning of a verb (predicate) without lexical or syntactic labeling.

We found that a transformer trained with a stripped-down sentence + predicate input could solve simple and complex problems. Challenging tasks...

Questions

1. Semantic Role Labeling (SRL) is a text-generation task. (True/False)
2. A predicate is a noun. (True/False)
3. A verb is a predicate. (True/False)
4. Arguments can describe who and what is doing something. (True/False)
5. A modifier can be an adverb. (True/False)
6. A modifier can be a location. (True/False)
7. A GPT-based model contains an encoder and decoder stack. (True/False)
8. A GPT-based SRL model has standard input formats. (True/False)
9. Transformers can solve any SRL task. (True/False)
10. ChatGPT can perform SRL better than any model. (True/False)

References

• Peng Shi and Jimmy Lin, 2019, Simple BERT Models for Relation Extraction and Semantic Role Labeling: https://arxiv.org/abs/1904.05255

Further reading

• Ce Zheng, Yiming Wang, and Baobao Chang, 2022, Query Your Model with Definitions in FrameNet: An Effective Method for Frame Semantic Role Labeling: https://arxiv.org/abs/2212.02036
• Emma Strubell and Andrew McCallum, 2018, Syntax Helps ELMo Understand Semantics: Is Syntax Still Relevant in a Deep Neural Architecture for SRL?: https://arxiv.org/abs/1811.04773

Join our community on Discord

Join our community’s Discord space for discussions with the authors and other readers:

https://www.packt.link/Transformers