How to open a local application using Cortana skills

Question

I've been trying to do this since this morning and it doesn't seem to be working for me.

The requirement is to have the user invoke cortana ana ask her to open an application - let's call it app1.

I created an azure bot based on the EchoBot and replaced the answering code with this:

protected override async Task
OnMessageActivityAsync(ITurnContext<IMessageActivity> turnContext,
CancellationToken cancellationToken)
    {
        await turnContext.SendActivityAsync(MessageFactory.Text($"Echo: 
                                Opening app1..."), cancellationToken);

        var startInfo = new ProcessStartInfo
        {
            FileName = @"D:\_Projects\xyz\app1.exe",
            UseShellExecute = false,
            CreateNoWindow = false,
        };

        var process = Process.Start(startInfo);
        var success = process != null && process.WaitForExit(30 * 10000);

        if (!success)
        {
            //process?.Kill();
            throw new ApplicationException("A timeout occurred during 
                 method execution. The service interface did not finish in a
                        timely fashion.");
        }
        var exitCode = process.ExitCode;
    }

This does work when executed locally after downloading the code from Azure. But it does not work when invoked from Cortana.

Edit: Local testing done using the Bot Framework Emulator (v4)

It simply prints the Opening App1 line and stands there. The debug window is as expected - useless.

Now I tried using a completely different technique I read somewhere, it consists of adding the application locally to the user\programs\ folder and invoking it from Cortana by saying Open app1.

The problem is, Cortana doesn't recognize the app at all. it just starts Edge and searches for app1 on bing.

I've seen a few videos regarding the cortana skill and in some of them launching a new app using done using a uwp application - But mine is actually an exe generated from python using auto-py-to-exe so this is not useful to me.

References: https://www.youtube.com/watch?v=h2L9KAWh5qs&t=2696s https://www.youtube.com/watch?v=6imjt5l7jXc

Is there a solution for this problem?

Answer 1

The reason this works locally and not when deployed is because your code executes Process.Start() on whatever machine is running the code (server side) and not on the machine of the person interacting with your code (client side). When testing locally, server and client are the same machine; not so when deployed.

It's possible, on a limited basis, to accomplish what you want.

Read the doc, Launch apps or websites from a Cortana skill.

Currently, Cortana supports a single action: LaunchUri. So, to launch an app, the app must have a uri protocol associated with it.

For example, to launch the Windows Map App, which uses bingmaps: protocol, you would use (Note: My example uses Bot Framework V4 code, whereas the docs I linked use V3):

var message = MessageFactory.Text("Launching app...");
message.ChannelData = JObject.FromObject(new {
     action = new { type = "LaunchUri", uri = "bingmaps:?where=Paris"}
});

await context.SendActivityAsync(message);

If you need to access a non-UWP app, you'd have to register your own URI scheme on your local machine in the registry:

  your-protocol-name/
    (Default)    "URL:your-protocol-name Protocol"
    URL Protocol ""
    shell/
      open/
        command/
          (Default) PathToExecutable