Problems after deployment

After an NLU system is developed and put into place in an application, it still requires monitoring. Once the system has reached an acceptable level of performance and has been deployed, it can be tempting to leave it alone and assume that it doesn’t need any more attention, but this is not the case. At the very least, the deployed system will receive a continuous stream of new data that can be challenging to the existing system if it is different from the training data in some way. On the other hand, if it is not different, it can be used as new training data. Clearly, it is better to detect performance problems from internal testing than to learn about them from negative customer feedback.

At a high level, we can think of new performance problems as either being due to a change in the system itself, or due to a change in the deployment context.

Changes in system performance due to system changes should be detected by testing before the new system...