Dialogflow - Repeat last sentence (voice) for Social Robot Elderly

Question

I am using actions-on-google and Dialogflow to build a social robot for Elderly.

I was wondering how I can easily repeat the last sentence when asked by the user ("repeat please") as often the Senior doesn't hear the sentence the first time.

One way would be to have repeat followup intents in Dialogflow but this is quite heavy since :

• you need to add one after each intent and I have many
• in a multi-user environment you need to keep track of the last sentence for every user ...

Another way would be be to take advantage of Dialogflow Contexts. As you send the message, you can also add that message to a context (for example, you can call it "last_message"). You can then have another Intent that takes as an input context the "last_message" context and, if triggered, uses the value saved in the context to repeat it.

• However, I still have the problem that I need to add a context to every intent I have, which are many.

Does anyone know how to accomplish this in a quicker way? I found this package but it is in JS and I need it Python: https://github.com/SysCoder/VoiceRepeater/pulls .

How do I implement this VoiceRepeater library? Do I put the code under fulfillment function 'repeat' I have made and mapped to an intent called 'repeat' that I have made which responds to utterances such as 'Sorry, could you repeat that'? Also, where do I install the VoiceRepeater library (code: npm install voice-repeater --save)?

Answer 1

Using Followup Intents is probably the wrong way to do this. As you note, it is way too heavy for more than a few Intents. It may be useful in certain circumstances if you want the "repeated" message to clarify the response in a different way, but in general, it isn't very useful. (It should also be noted that Followup Intents use Contexts, but in a different way than discussed below.)

You don't need to add the Context to the UI as part of the Outgoing Context - you would set this as part of your Fulfillment. It would include a parameter that either contained exactly what you said, or the information you needed to recreate what you said (possibly in a different form, if appropriate). In your "repeat" Intent, you'd read the value that you had saved in this Context, and send it as the output again. If you're using SSML, you may wish to change the speed or volume, if that is appropriate.

Update based on new questions

The readme for VoiceRepeater has the basics of what you need to do to use it, but it does assume a little familiarity with Node. But in general, yes, you install it the way you describe, setup an Intent that captures requests to repeat, and registers a handler function (repeatLastStatement(app) in the readme) that handles the Intent to send a reply through voiceRepeater.lastPromptWithPrefix().

It also may assume you're using version 1 of the actions-on-google library. I haven't dug too deeply into the code, but it looks like it replaces the library's ask function with its own, and I'm not sure how well that works with version 2 of the actions-on-google library.

Unlike Voice Repeater, multivocal doesn't require you to register handlers specifically since it tries to hide as much boilerplate under the covers. You just need to define the replies that you might want it to use. It uses the Context scheme I outline above to store responses and make them available when the user asks for it to be repeated.

There aren't any videos on using multivocal, but the simple example does include the configuration illustrating how to configure responses for the "multivocal.repeat" Intent. While VoiceRepeater works with the actions-on-google library, multivocal is a complete replacement, offering a more template-based approach to building fulfillment.

However, neither of these directly help you if you want to implement it for Python. But if you look at the source for VoiceRepeater, you can get a sense for how to implement it yourself in Python.

The key bit is on line 47 where it saves the reply in a context. (It also saves the reply with a prefix message.) It then calls the original function that would send the reply:

  app.setContext("last_prompt", 100,
    {
      "last_prompt": textToSpeech,
      "prefixed_last_prompt": repeatPrefix + lastStatement,
    });
  originalAsk(response);

Later, in the call to lastPromptWithPrefix(), it uses the contents of the Context to send a reply.

  lastPromptWithPrefix() {
     return this.app.getContext("last_prompt") !== null
         ? this.app.getContextArgument("last_prompt", "prefixed_last_prompt").value
         : "um....I don't remember what I said!";
  }