Fine-tuning the BERT model for sentence-pair regression

The regression model is considered to be for classification, but the last layer only contains a single unit. This is not processed by softmax logistic regression but normalized. To specify the model and put a single-unit head layer at the top, we can either directly pass the num_labels=1 parameter to the BERT.from_pre-trained() method or pass this information through a Config object. Initially, this needs to be copied from the config object of the pre-trained model, as follows:

from transformers import DistilBertConfig, DistilBertTokenizerFast, DistilBertForSequenceClassification
model_path='distilbert-base-uncased'
config = DistilBertConfig.from_pre-trained(model_path, num_labels=1)
tokenizer = DistilBertTokenizerFast.from_pre-trained(model_path)
model = \
DistilBertForSequenceClassification.from_pre-trained(model_path, config=config)

Well, our pre-trained model has a single-unit head layer thanks to the num_labels...