I followed this SO answer, https://stackoverflow.com/a/65736139/3970346
Since this is a NativeScript app the steps are,
Create a ids.xml
file under app/App_Resources/Android/src/main/res/values folder,
<?xml version="1.0" encoding="utf-8"?>
<resources>
<item type="id" name="overflowActionButton"/>
</resources>
Then in the styles.xml in the same directory add,
<style name="Widget.ActionButton.Overflow" parent="Widget.AppCompat.ActionButton.Overflow">
<item name="android:id">@id/overflowActionButton</item>
<item name="android:contentDescription">"overflowActionButton"</item>
<item name="android:tint">@color/white</item>
</style>
Finally set the style on the AppThemeBase like this,
<!-- theme to use AFTER launch screen is loaded-->
<style name="AppThemeBase" parent="Theme.AppCompat.Light.NoActionBar">
<item name="toolbarStyle">@style/NativeScriptToolbarStyle</item>
<item name="colorPrimary">@color/ns_primary</item>
<item name="colorPrimaryDark">@color/ns_primaryDark</item>
<item name="colorAccent">@color/ns_accent</item>
<!-- the line below -->
<item name="actionOverflowButtonStyle">@style/Widget.ActionButton.Overflow</item>
</style>
After this you can tap on the overflow menu in Detox like this,
await element(by.label('overflowActionButton')).tap();
Thanks to @Leri Gogsadze for pointing me in the right direction