Finding the coordinates of a button or text on the screen using the accessibility service

Question

Is it possible to use the accessibility service to find the coordinates of a button or the text on the screen, then simulate tapping on that button or text field, but without touching the field, like you need to touch the field in Android TalkBack to hear reading?

For example, is it possible to use the accessibility service like this:

1. Send the intent from the activity to the accessibility service with some extras for different tasks.

2. When receiving an intent with a certain extra string, the service will open the developer options.

3. Without touching the text "Running services", the accessibility service should find that text field (or button) coordinates in the developer options and simulate a tap on it.

Is that possible on Android or not?

Answer 1

TL;DR; I think I've written a service you might find interesting

Some points to keep in mind:

1. You need to enable the service and have it running.
2. Google has forbidden accessibility services to be released on the Play Store if they are not intended for helping people with disabilities
3. Accessibility services can only "see" what is on the screen currently - this a problem with RecyclerViews as the ViewHolder is reused for different elements as the user scrolls.

However you can do this in an accessibility service:

    // To find a node by *fully qualified id*
    rootInActiveWindow.findAccessibilityNodeInfosByViewId("id criteria")

    // To find a node by text criteria
    rootInActiveWindow.findAccessibilityNodeInfosByText("text criteria")

When searching by id criteria:

Finds AccessibilityNodeInfos by the fully qualified view id's resource name where a fully qualified id is of the from "package:id/id_resource_name". For example, if the target application's package is "foo.bar" and the id resource name is "baz", the fully qualified resource id is "foo.bar:id/baz".

Note: The primary usage of this API is for UI test automation and in order to report the fully qualified view id if an AccessibilityNodeInfo the client has to set the AccessibilityServiceInfo#FLAG_REPORT_VIEW_IDS flag when configuring the AccessibilityService.

When searching by text criteria:

Finds AccessibilityNodeInfos by text. The match is case insensitive containment. The search is relative to this info i.e. this info is the root of the traversed tree.

Note for both:

Note: If this view hierarchy has a SurfaceView embedding another view hierarchy via SurfaceView#setChildSurfacePackage, there is a limitation that this API won't be able to find the node for the view on the embedded view hierarchy. It's because views don't know about the embedded hierarchies. Instead, you could traverse all the children to find the node.

To click on an element:

    fun click(long: Boolean = false) {
        someAccessibilityNodeInfoObject.performAction(
            if (long) ACTION_LONG_CLICK 
            else ACTION_CLICK
        )
    }

If you are really passionate about getting the screen coordinates of a node you can always use the getBoundsInScreen method, but then you have to do all the math and stuff and that's harder than it needs to be. But I did it in bash when doing a adb uiautomator dump and parsing the XML, stripping the bounds for the node I want, just be wary that using uiautomator dump disables the accessibility service.

MIDX= START_X + ((END_X - START_X) / 2)
MIDY= START_Y + ((END_Y - START_Y) / 2)