Search code examples
androidkotlincanvasondrawaccessibilityservice

How Google TalkBack service draws Rectangle over buttons/texts/images/etc.?

Before you start typing I would say that you were going to write down the documentation of AccessibilityService, AccessibilityNodeInfo, etc.

I've implemented my AccessibilityService correctly and I am even getting the AccessibilityEvent-s correctly.

I am only concerned about how to draw Rectangle around the view by user's clicks.

What I have done so far regarding to this:

  • Created custom view class which implemented ViewGroup
  • I added that CustomView (class I wrote) as a View with WindowManager, byt calling .addView() method
  • I am drawing Rectangle on Canvas by calling canvas.drawRect() from onDraw() overloaded method
  • When adding view on the screen I use WindowManager.LayoutParams.FLAG_WATCH_OUTSIDE_TOUCH flag and WindowManager.LayoutParams.TYPE_ACCESSIBILITY_OVERLAY type for parameters to pass into .addView() method

You may wonder what's the problem here..

Here They are:

  1. When adding new CustomView with WindowManager service, the whole new View covers the screen and other buttons are not accessible. ( I literally can't click anything except the newly added layer, I think it's getting drawn on top of everything and everything becomes unclickable)
  2. adding custom view as a layer instead of drawing directly around some View, causes (or not, I don't know exactly) event.source.getBoundsInScreen(rect) in onAccessibilityEvent(event: AccessibilityEvent) to store wrong directions of button clicked by user. (When I draw the same Rectangle with canvas.drawRect(rect, mpaint) the rectangle isn't show exactly around the view, some coordinates are mistaken)

I want to make my AccessibilityService draw Rectangle like Google TalkBack does it.

The Google TalkBack draws PERFECT Rectangle around the View, but mine doesn't, unfortunately.

  • Here are the screenshots of my app drawing the rectangle: (clicked on '9')

mine screen

  • Here is the screenshot of Google TalkBack clicking '9' button:

talkbacks screen

Thanks.

Solution

  • The magic is that TalkBack doesn't draw the box. It's actually drawn in the app process. There's code in ViewRootImpl that handles it. This functionality can only be used when Touch Exploration is enabled, which make it unusable for anything not interested in a user experience very similar to TalkBack's.

    Anticipating the follow-up, no we don't plan to open this up. The overlay, while offsets are always fiddly, is a much more scalable approach to helping services highlight content.