I experienced multiple outages with our Hazelcast setup (Hazelcast embedded within Payara) in situation, where non-hazelcast workload of the process caused GC overhead errors.
During that time Hazelcast responded to heartbeats, however, did not respond to actual requests that fetched data. This caused system to stall, even if backup of the data was available.
Therefore I want to ask if there is a way to kick off a member out of cluster if it stops responding to other kinds of requests than just heartbeat.
You might want to consider using hazelcast in "Lite" mode on Payara server instances. If you turn on "lite" mode on the instances which might face ecessive GC, these instances should not affect Hazelcast performance, as those nodes would not store any data for the hazelcast cluster.
However, it is necessary to complement this solution with additional Full instances, which would keep the data. These additional instances should not run any application of yours and only function as a separate hazelcast node. The section Focused on throughput and performance in my blog post describes some scenarios how to use "Lite" Payara cluster members (backed by Lite hazelcast members).
To summarize - if you turn some nodes into "Lite" and for every "Lite" node you add an additional "Full" node that runs just by itself, with no deployed applications, you would separate the memory used by your applications and hazelcast into separate JVMs. GC on a JVM that is a "Lite" node and contains no data for the hazelcast cluster, should not affect Hazelcast performance.