To request a user for granting "Allow display over other apps" permission, one can send an Intent with Settings.ACTION_MANAGE_OVERLAY_PERMISSION action. For Android versions before R, (API version < 30) it goes directly to this specific setting in app setting page, but for API 30, it goes to "Display over other apps" permission page which is very bad for user experience.
I read that it's an intended behavior, but is there a workaround for this? For my screenshot app which needs to display a capture overlay button, what alternatives exist?
I read that it's an intended behavior, but is there a workaround for this?
I am not aware of any.
what alternatives exist?
You could use bubbles on Android 11+.
Long-term, I expect that overlays are going to be removed as an option. They are already limited in some environments (e.g., Android One), and there has been a steady move to eliminate these sorts of things (e.g., custom toasts). Unless Google comes up with something else, my guess is that apps that use an overlay for a trigger mechanism — which sounds like your scenario — will wind up migrating to bubbles or simply persistent notifications.
Note that bubbles have some UI beyond the bubble itself — tapping the bubble displays an activity in a pop-up window. That might or might not fit your use case.
And, FWIW, this sample module demonstrates how to show a bubble.