How does Appium/UiAutomator actually see composables?

Question

I have been rewriting my app to Jetpack Compose, and I already have existing Appium tests. After rewriting (using testTag modifiers to replace what used to be resource IDs) all of my tests work fine. So I am not asking for help with getting things working, but I am trying to understand how it works. When I look at the App Source tab in Appium, I see this:

But this screen is just an Activity with a ComposeView created with the setContent extension. How does Appium see a Column with a verticalScroll modifier as a ScrollView, and a TextField as an EditText, etc? I know Compose is not just creating Views under the hood, so where is this mapping being done? I tried searching the UiAutomator and Appium source code and could not find the right code.

Anyone out there have any expertise here to help me understand?

Answer 1

Appium uses UiAutomatorViewer, which you can find in Android\Sdk\tools\bin.

We can look at the source to find ScreenshotAction, which uses DumpCommand, which dumps the view as XML using AccessibilityNodeInfoDumper.

Every composable has AccessibilityNodeInfo, where the className property is the same as the XML equivalent class name.

You can find at least some of the "mapping" from compose to xml class names in AndroidComposeViewAccessibilityDelegateCompat.

If the composable has vertical scrolling, the className will be android.widget.ScrollView, like this:

val yScrollState = semanticsNode.config.getOrNull(SemanticsProperties.VerticalScrollAxisRange)
val scrollAction = semanticsNode.config.getOrNull(SemanticsActions.ScrollBy)
if (yScrollState != null && scrollAction != null) {
   info.className = "android.widget.ScrollView"
}

A lot of the mapping is also based on Role:

        semanticsNode.config.getOrNull(SemanticsProperties.Role)?.let {
            when (it) {
                Role.Button -> info.className = "android.widget.Button"
                Role.Checkbox -> info.className = "android.widget.CheckBox"
                Role.Switch -> info.className = "android.widget.Switch"
                Role.RadioButton -> info.className = "android.widget.RadioButton"
                Role.Tab -> info.roleDescription = AccessibilityRoleDescriptions.Tab
                Role.Image -> info.className = "android.widget.ImageView"
            }
        }

That's one of the reasons it's important to set a Role for you custom composables. Like:

@Composable 
fun CustomButton() {
    Box(
        modifier = Modifier.semantics {
            role = Role.Button
        }
    )
}