Search code examples
androidxamarintalkbackd-pad

Handle Talkback in a Xamarin app using a virtual DPAD

I have a Xamarin app that was not meant to handle the talkback functionality of android, because for it to work well it had to be build in a specific way.

My app is a little order, and I simply can't make a do-over of the whole thing.

So, what is happening? My Xamarin app is made with non-native libs, that are not supported by the Talkback, so, when the user turns on the Talkback functionality the app effectively stops receiving the DPAD events since they are handled by the systems Accessibility Service.

That service, gets the events, and tries to handle them within my app, but, since my components are non-native, the system does not recognize them and the DPAD is wasted, hence, the illusion that the DPADs are not working.

So, what do you have to do if you just want to handle the DPADs (and nothing else) yourself with Talkback on?

The answer to this post will contain the code that describes the following behavior:

1. The talkback wont be able to 'talk' about your components

2. The DPAD events will be handled by an Accessibility Delegate

3. A virtual DPAD will handle the navigation

4. The green rectangle used for focus will be disabled, since you wont need it anyway

5. The app will look exactly the same with Talkback on and off

This post was made for educational purposes, since I had a hard time coming up with the solution, and hope the next guy finds it helpfull.

Solution

  • The first step is to create a class that inherits the AccessibilityDelegateCompat in order to create our own Accessibility Service.

    class MyAccessibilityHelper : AccessibilityDelegateCompat
{
    const string Tag = "MyAccessibilityHelper";
    const int ROOT_NODE = -1;
    const int INVALID_NODE = -1000;
    const string NODE_CLASS_NAME = "My_Node";

    public const int NODE_UP = 1;
    public const int NODE_LEFT = 2;
    public const int NODE_CENTER = 3;
    public const int NODE_RIGHT = 4;
    public const int NODE_DOWN = 5;

    private class MyAccessibilityProvider : AccessibilityNodeProviderCompat
    {
        private readonly MyAccessibilityHelper mHelper;

        public MyAccessibilityProvider(MyAccessibilityHelper helper)
        {
            mHelper = helper;
        }

        public override bool PerformAction(int virtualViewId, int action, Bundle arguments)
        {
            return mHelper.PerformNodeAction(virtualViewId, action, arguments);
        }

        public override AccessibilityNodeInfoCompat CreateAccessibilityNodeInfo(int virtualViewId)
        {
            var node = mHelper.CreateNode(virtualViewId);
            return AccessibilityNodeInfoCompat.Obtain(node);
        }
    }

    private readonly View mView;
    private readonly MyAccessibilityProvider mProvider;
    private Dictionary<int, Rect> mRects = new Dictionary<int, Rect>();
    private int mAccessibilityFocusIndex = INVALID_NODE;

    public MyAccessibilityHelper(View view)
    {
        mView = view;
        mProvider = new MyAccessibilityProvider(this);
    }

    public override AccessibilityNodeProviderCompat GetAccessibilityNodeProvider(View host)
    {
        return mProvider;
    }

    public override void SendAccessibilityEvent(View host, int eventType)
    {
        Android.Util.Log.Debug(Tag, "SendAccessibilityEvent: host={0} eventType={1}", host, eventType);
        base.SendAccessibilityEvent(host, eventType);
    }

    public void AddRect(int id, Rect rect)
    {
        mRects.Add(id, rect);
    }

    public AccessibilityNodeInfoCompat CreateNode(int virtualViewId)
    {
        var node = AccessibilityNodeInfoCompat.Obtain(mView);
        if (virtualViewId == ROOT_NODE)
        {
            node.ContentDescription = "Root node";
            ViewCompat.OnInitializeAccessibilityNodeInfo(mView, node);
            foreach (var r in mRects)
            {
                node.AddChild(mView, r.Key);
            }
        }
        else
        {
            node.ContentDescription = "";
            node.ClassName = NODE_CLASS_NAME;
            node.Enabled = true;
            node.Focusable = true;
            var r = mRects[virtualViewId];
            node.SetBoundsInParent(r);
            int[] offset = new int[2];
            mView.GetLocationOnScreen(offset);
            node.SetBoundsInScreen(new Rect(offset[0] + r.Left, offset[1] + r.Top, offset[0] + r.Right, offset[1] + r.Bottom));
            node.PackageName = mView.Context.PackageName;
            node.SetSource(mView, virtualViewId);
            node.SetParent(mView);
            node.VisibleToUser = true;
            if (virtualViewId == mAccessibilityFocusIndex)
            {
                node.AccessibilityFocused = true;
                node.AddAction(AccessibilityNodeInfoCompat.ActionClearAccessibilityFocus);
            }
            else
            {
                node.AccessibilityFocused = false;
                node.AddAction(AccessibilityNodeInfoCompat.FocusAccessibility);
            }
        }
        return node;
    }

    private AccessibilityEvent CreateEvent(int virtualViewId, EventTypes eventType)
    {
        var e = AccessibilityEvent.Obtain(eventType);
        if (virtualViewId == ROOT_NODE)
        {
            ViewCompat.OnInitializeAccessibilityEvent(mView, e);
        }
        else
        {
            var record = AccessibilityEventCompat.AsRecord(e);
            record.Enabled = true;
            record.SetSource(mView, virtualViewId);
            record.ClassName = NODE_CLASS_NAME;
            e.PackageName = mView.Context.PackageName;
        }
        return e;
    }

    public bool SendEventForVirtualView(int virtualViewId, EventTypes eventType)
    {
        if (mView.Parent == null)
            return false;
        var e = CreateEvent(virtualViewId, eventType);
        return ViewParentCompat.RequestSendAccessibilityEvent(mView.Parent, mView, e);
    }

    public bool PerformNodeAction(int virtualViewId, int action, Bundle arguments)
    {
        if (virtualViewId == ROOT_NODE)
        {
            return ViewCompat.PerformAccessibilityAction(mView, action, arguments);
        }
        else
        {
            switch (action)
            {
                case AccessibilityNodeInfoCompat.ActionAccessibilityFocus:
                    if (virtualViewId != mAccessibilityFocusIndex)
                    {
                        if (mAccessibilityFocusIndex != INVALID_NODE)
                        {
                            SendEventForVirtualView(mAccessibilityFocusIndex, EventTypes.ViewAccessibilityFocusCleared);
                        }
                        mAccessibilityFocusIndex = virtualViewId;
                        mView.Invalidate();
                        SendEventForVirtualView(virtualViewId, EventTypes.ViewAccessibilityFocused);
                        // virtual key event                            
                        switch (virtualViewId)
                        {
                            case NODE_UP:
                                HandleDpadEvent(Keycode.DpadUp);
                                break;
                            case NODE_LEFT:
                                HandleDpadEvent(Keycode.DpadLeft);
                                break;
                            case NODE_RIGHT:
                                HandleDpadEvent(Keycode.DpadRight);
                                break;
                            case NODE_DOWN:
                                HandleDpadEvent(Keycode.DpadDown); 
                                break;
                        }
                        // refocus center
                        SendEventForVirtualView(NODE_CENTER, EventTypes.ViewAccessibilityFocused);
                        return true;
                    }
                    break;
                case AccessibilityNodeInfoCompat.ActionClearAccessibilityFocus:
                    mView.RequestFocus();
                    if (virtualViewId == mAccessibilityFocusIndex)
                    {
                        mAccessibilityFocusIndex = INVALID_NODE;
                        mView.Invalidate();
                        SendEventForVirtualView(virtualViewId, EventTypes.ViewAccessibilityFocusCleared);
                        return true;
                    }
                    break;
            }
        }
        return false;
    }

    private void HandleDpadEvent(Keycode keycode)
    {
       //Here you know what DPAD was pressed
       //You can create your own key event and send it to your app
       //This code depends on your own application, and I wont be providing the code
       //Note, it is important to handle both, the KeyDOWN and the KeyUP event for it to work
    }
}

    Since the code is a bit large, I'll just explain the crutal parts. Once the talkback is active, the dictionary (from our view bellow) will be used to create a virtual tree node of our virtual DPAD. With that in mind, the function PerformNodeAction will be the most important one.

    It handles the actions once a virtual node was focused by the Accessibility system, based on the provided id of the virtual element, there are two parts, the first one is the ROOT_NODE, which is the view iteslf that contains our virtual dpad, which for the most part can be ignored, but the seond part is where the handling is done.

    The second part is where the actions ActionAccessibilityFocus and ActionClearAccessibilityFocus are handled. The two of witch are both important, but the first one is where we can finally handle our virtual dpad.

    What is done here is that with the provided virtual ID from the dictionary, we know which DPAD was selected (virtualViewId). Based on the selected DPAD, we can perform the action we want in the HandleDpadEvent function. What is important to notice, is that after we handle the selecteds DPAD event, we will refocus our CENTER node, in order to be ready to handle the next button press. This is very important, since, you dont want to find yourself in a situation where you go DOWN, and then UP, just for the virtual dpad to focus the CENTER pad.

    So, I'll epeat myself, the refocusing of the CENTER pad after the previous' DPAD event was handled needs to be done in order for us to know EXACTLY where we will be after the next DPAD button was pressed!

    There is one function that I wont post here, since the code for it is very specific for my app, the function is HandleDpadEvent, there you must create a keydown and a keyup event and send it to your main activity where the function onKeyDown/Up will be triggered. Once you do that, the delegate is done.

    And once the Delegate is done, we have to make our view like this:

    /**
* SimplestCustomView
*/
public class AccessibilityHelperView : View
{
    private MyAccessibilityHelper mHelper;

    Dictionary<int, Rect> virtualIdRectMap = new Dictionary<int, Rect>();

    public AccessibilityHelperView(Context context) :
        base(context)
    {
        Init();
    }

    public AccessibilityHelperView(Context context, IAttributeSet attrs) :
        base(context, attrs)
    {
        Init();
    }

    public AccessibilityHelperView(Context context, IAttributeSet attrs, int defStyle) :
        base(context, attrs, defStyle)
    {
        Init();
    }

    public void Init()
    {
        this.SetFocusable(ViewFocusability.Focusable);
        this.Focusable = true;
        this.FocusedByDefault = true;

        setRectangle();

        mHelper = new MyAccessibilityHelper(this);
        ViewCompat.SetAccessibilityDelegate(this, mHelper);
        foreach (var r in virtualIdRectMap)
        {
            mHelper.AddRect(r.Key, r.Value);
        }
    }

    private void setRectangle()
    {
        virtualIdRectMap.Add(MRAccessibilityHelper.NODE_CENTER, new Rect(1, 1, 2, 2));
        virtualIdRectMap.Add(MRAccessibilityHelper.NODE_LEFT, new Rect(0, 1, 1, 2));
        virtualIdRectMap.Add(MRAccessibilityHelper.NODE_UP, new Rect(1, 0, 2, 1));
        virtualIdRectMap.Add(MRAccessibilityHelper.NODE_RIGHT, new Rect(2, 1, 3, 2));
        virtualIdRectMap.Add(MRAccessibilityHelper.NODE_DOWN, new Rect(1, 2, 2, 3));
    }

    protected override void OnDraw(Canvas canvas)
    {
        base.OnDraw(canvas);
    }
}

    That view looks like this:

    Virtual DPad

    What is to notice?

    1. The size of the node pads is in pixels, and they will be found on the top left corner of your app.

    2. They are set to that single pixel size, because the Talkback functionality would otherwise select the first node pad that was added to the dictionary with a green rectangle (thats standard behavior for talkback)

    3. All the rectangles in the view are added to a dictionary that will be used in our own Accessibility Delegate, to mention here is that the CENTER pad was added first, and therefor will be in focus once the talkback is activated by default

    4. The Init function

    The Init function is crutial for this, there we will create our view, and set some talkback parameters nessessary for our virtual dpad to be recognized by the systems own Accessibility Service.

    Also, there will our Accessibility Delegate be initialized and our dictionary with all the created DPADs.

    Ok, so far, we made a Delegate and a View, I placed them both in the same file, so they can see each other. But it is not a must.

    So what now? We must add the AccessibilityHelperView to our app, in the MainActivity.cs file

    AccessibilityHelperView mAccessibilityHelperView;

    In the OnCreate function, you can add the following code to initiate the view:

    mAccessibilityHelperView = new AccessibilityHelperView(this);

    In the OnResume function, you can check if the talkback is on or off, based on the result, you can add or remove the mAccessibilityHelperView from your mBackgroundLayout(AddView, and RemoveView).

    The OnResume function should look like this:

     if (TalkbackEnabled && !_isVirtualDPadShown)
 {
     mBackgroundLayout.AddView(mAccessibilityHelperView);
     _isVirtualDPadShown = true;
 }
 else if (!TalkbackEnabled && _isVirtualDPadShown)
 {
      mBackgroundLayout.RemoveView(mAccessibilityHelperView);
      _isVirtualDPadShown = false;
 }

    The TalkbackEnabled variable is a local one that checks if the Talkback service is on or off, like this:

     public bool TalkbackEnabled 
 {
        get
        {
            AccessibilityManager am = MyApp.Instance.GetSystemService(Context.AccessibilityService) as AccessibilityManager;
            if (am == null) return false;

            String TALKBACK_SETTING_ACTIVITY_NAME = "com.android.talkback.TalkBackPreferencesActivity";
            var serviceList = am.GetEnabledAccessibilityServiceList(FeedbackFlags.AllMask);
            foreach (AccessibilityServiceInfo serviceInfo in serviceList)
            {
                String name = serviceInfo.SettingsActivityName;
                if (name.Equals(TALKBACK_SETTING_ACTIVITY_NAME))
                {
                    Log.Debug(LogArea, "Talkback is active");
                    return true;
                }
            }
            Log.Debug(LogArea, "Talkback is inactive");
            return false;
        }
 }

    That should be all you need to make it work.

    Hope I could help you out.