The IBM Watson Conversation service offers as part of its UI "Improve". It has the ability to see recent messages, their classification and then to reclassify or mark as irrelevant.
In the REST API for that conversation service there functions to list and create counterexamples and examples. I looked at the Python SDK and that API is supported, too. How do I programmatically turn recent user conversation (messages, user input) into examples or counterexamples? What API function do I need to use?
You need to use the logging API referenced in the Workspace API.
The logging uses elastic search if you need to filter it down.
It is not going to give you the breakdown similar to the improve tab though. So you will need to programatically track what you want to capture and add.
I would be very wary though of having any kind of automation updating your training. Proper training should only be done by a subject matter expert. Automating it is likely to pollute the training.
Another thing to avoid is don't add correct answers to the training if Conversation got it correct. It's redundant and also means further testing to see that it doesn't impact your existing model.