Unable to get Twilio sms status callbacks when sending proactive message

I'm trying to track the sms delivery status of the messages I send using the bot framework. I'm using Twilio, and sending proactive messages. Right now I'm trying to do so with twilio status callbacks

This is similar to this question, I tried that approach but I couldn't get it to work. I've added my url on the TwiML app and it is not firing. I have double and triple checked, and I suspect this url is somehow ignored or not going through with my current set up. I don't get any callbacks on the proactive message nor on the replies the bot sends to the user. However the flow works fine and I can reply and get proper responses from the bot. Edit: calling this "approach 1"

approach 2: I've also tried this doing some light modifications on Twilio adapter, to be able to add my callback just before create message. (I changed it so it uses a customized client wrapper that adds my callback url when creating the twilio resource) This does work, partially: when I reply a message from my bot, I get the status callbacks. But as the proactive message is sent using the default adapter, I don't get a callback on the initial message.

approach 3: Finally, I also tried using the TwilioAdapter when sending the proactive message but for some reason as soon as I send an activity, the TurnContext is disposed, so I can't save the state or do any subsequent actions. This leads me to believe twilio adapter is not intended to be used this way (can't be used on proactive messages), but I'm willing to explore this path if necessary.

Here is the modified Twilio Adapter:

public class TwilioAdapterWithErrorHandler : TwilioAdapter
{
    private const string TwilioNumberKey = "TwilioNumber";
    private const string TwilioAccountSidKey = "TwilioAccountSid";
    private const string TwilioAuthTokenKey = "TwilioAuthToken";
    private const string TwilioValidationUrlKey = "TwilioValidationUrl";

    public TwilioAdapterWithErrorHandler(IConfiguration configuration, ILogger<TwilioAdapter> logger, TwilioAdapterOptions adapterOptions = null)
        : base(
        new TwilioClientWrapperWithCallback(new TwilioClientWrapperOptions(configuration[TwilioNumberKey], configuration[TwilioAccountSidKey], configuration[TwilioAuthTokenKey], new Uri(configuration[TwilioValidationUrlKey]))), adapterOptions, logger)
    {
        OnTurnError = async (turnContext, exception) =>
        {
            // Log any leaked exception from the application.
            logger.LogError(exception, $"[OnTurnError] unhandled error : {exception.Message}");

            Task[] tasks = {
                // Send a message to the user
                turnContext.SendActivityAsync("We're sorry but this bot encountered an error when processing your answer."),
                // Send a trace activity, which will be displayed in the Bot Framework Emulator
                turnContext.TraceActivityAsync("OnTurnError Trace", exception.Message, "https://www.botframework.com/schemas/error", "TurnError")
            };

            Task all = Task.WhenAll(tasks); //task with the long running tasks

            await Task.WhenAny(all, Task.Delay(5000)); //wait with a timeout
        };
    }
}

Modified client Wrapper:

public class TwilioClientWrapperWithCallback : TwilioClientWrapper
{
    public TwilioClientWrapperWithCallback(TwilioClientWrapperOptions options) : base(options) { }

    public async override Task<string> SendMessageAsync(TwilioMessageOptions messageOptions, CancellationToken cancellationToken)
    {
        var createMessageOptions = new CreateMessageOptions(messageOptions.To)
        {
            ApplicationSid = messageOptions.ApplicationSid,
            MediaUrl = messageOptions.MediaUrl,
            Body = messageOptions.Body,
            From = messageOptions.From,
        };

        createMessageOptions.StatusCallback = new System.Uri("https://myApp.ngrok.io/api/TwilioSms/SmsStatusUpdated");

        var messageResource = await MessageResource.CreateAsync(createMessageOptions).ConfigureAwait(false);
        return messageResource.Sid;
    }
}

Finally, here's my summarized code that sends the proactive message:

[HttpPost("StartConversationWithSuperBill/{superBillId:long}")]
[HttpPost("StartConversationWithSuperBill/{superBillId:long}/Campaign/{campaignId:int}")]
public async Task<IActionResult> StartConversation(long superBillId, int? campaignId)
{
    ConversationReference conversationReference = this.GetConversationReference("+17545517768");

    //Start a new conversation.
    await ((BotAdapter)_adapter).ContinueConversationAsync(_appId, conversationReference, async (turnContext, token) =>
    {
        await turnContext.SendActivityAsync("proactive message 1");
        //this code was edited for brevity. Here I would start a new dialog that would cascade into another, but the end result is the same, as soon as a message is sent, the turn context is disposed.
        await turnContext.SendActivityAsync("proactive message 2"); //throws ObjectDisposedException
    }, default(CancellationToken));

    var result = new { status = "Initialized fine!" };
    return new JsonResult(result);
}

private ConversationReference GetConversationReference(string targetNumber)
{
    string fromNumber = "+18632704234";
    return new ConversationReference
    {
        User = new ChannelAccount { Id = targetNumber, Role = "user" },
        Bot = new ChannelAccount { Id = fromNumber, Role = "bot" },
        Conversation = new ConversationAccount { Id = targetNumber },
        //ChannelId = "sms",
        ChannelId = "twilio-sms", //appparently when using twilio adapter we need to set this. if using TwiML app and not using Twilio Adapter, use the above. Otherwise the frameworks interprets answers from SMS as new conversations instead.
        ServiceUrl = "https://sms.botframework.com/",
    };
}

(I can see that I could just call create conversation reference and do two callbacks, one for each message, but in my actual code I'm creating a dialog that sends one message and then invokes another dialog that starts another message)

Edit 2:

Some clarifications:

On approach 2, I'm using two adapters, as suggested by code sample and documentation on using twilio adapter. The controller that starts the proactive message uses an instance of a default adapter (similar to this one), and TwilioController (the one that gets the twilio incoming messages) uses TwilioAdapterWithErrorHandler.

On approach 3, I excluded the default adapter, and both controllers use TwilioAdapterWithErrorHandler.

Edit 3:

Here's a small repo with the issue.

Solution

I found a fix for this problem, around approach 3, by changing the overload I use for ContinueConversation. Replace this :

    //Start a new conversation.
    await ((BotAdapter)_adapter).ContinueConversationAsync(_appId, conversationReference, async (turnContext, token) =>

With this:

    //Start a new conversation.
    var twilioAdapter = (TwilioAdapterWithErrorHandler)_adapter;
    await twilioAdapter.ContinueConversationAsync(_appId, conversationReference, async (context, token) =>

This way, the context is not disposed, an I can use the twilio adapter for the proactive message and have status callbacks on all messages.