Android Macrobenchmark 1.2.4: StartupMode.COLD breaks the benchmarks. Inside measure block UiAutomator can't find any objects

Question

I have added macrobenchmark module with StartupTimingMetric and FrameTimingMetric metrics long time ago and my setup was working fine. Since then I have not been running benchmarks due to some reasons. Recently, had time to go back and update the user flows and run the benchmarks again. But my FrameTimingMetric benchmarks started to fail at step of running measureBlock. UiAutomator couldn't find any object by resource id inside of measureBlock but could see all of them in setupBlock. I'm using Compose and necessary composables have testTag added.

It was working fine a year ago when I have added benchmarks initially, but now they stopped. In the end I managed to fix the problem by removing startupMode = StartupMode.COLD. I have no idea why this flag is affecting the visibility of composables in measureBlock only, while setupBlock works just fine.

Could someone help me understand why this is happening?

// NOT working case

        rule.measureRepeated(
            packageName = InstrumentationRegistry.getArguments().getString("targetAppId")
                ?: throw Exception("targetAppId not passed as instrumentation runner arg"),
            metrics = listOf(FrameTimingMetric()),
            compilationMode = compilationMode,
            startupMode = StartupMode.COLD,
            iterations = 10,
            setupBlock = {
                pressHome()
                startActivityAndWait()
                // other setup methods
                // "MyList" is visible here and list var is NOT null
                // val list = device.findObject(By.res("MyList"))
            },
            measureBlock = {
                // "MyList" is NOT visible here and list var IS null
                val list = device.findObject(By.res("MyList"))
            }
        )

// Working case

        rule.measureRepeated(
            packageName = InstrumentationRegistry.getArguments().getString("targetAppId")
                ?: throw Exception("targetAppId not passed as instrumentation runner arg"),
            metrics = listOf(FrameTimingMetric()),
            compilationMode = compilationMode,
            // startupMode = StartupMode.COLD,      // <- REMOVE this line
            iterations = 10,
            setupBlock = {
                pressHome()
                startActivityAndWait()
                // other setup methods
                // "MyList" is visible here and list var is NOT null
                // val list = device.findObject(By.res("MyList"))
            },
            measureBlock = {
                // "MyList" is visible here and list var is NOT null
                val list = device.findObject(By.res("MyList"))
            }
        )

Answer 1

If StartupMode.COLD is used, the app process is killed between the execution of setupBlock and measureBlock to allow for app preparation without starting the process.

If you need the process to remain active, use StartupMode.WARM, which restarts activities without restarting the process, or set startupMode to null and call killProcess() within the setupBlock.

You can subscribe to b/278214396 to get more updates about this in the future.