Certificate validation in general requires asynchronous operations such as: OCSP/CRL fetch.
On the other hand, the callback from SSL_CTX_set_cert_verify_callback expects a synchronous result:
1 (success) / 0 (failure). (reference).
I find it kind of odd that any validation of this type is expected to be synchronous.
Is there an alternative API for certificate chain validation that supports asynchronous operations?
In theory I think you should be able to use the OpenSSL ASYNC API to do this. Normally this is used by asynchronous capable engines, but I don't see why it wouldn't also work in applications.
Firstly you would need to put your SSL object into ASYNC mode using the function call SSL_set_mode() and specifying SSL_MODE_ASYNC. You can also do this at the SSL_CTX level using SSL_CTX_set_mode(). See:
https://www.openssl.org/docs/man1.1.1/man3/SSL_set_mode.html
Once that is done you need to ensure your application is prepared to handle the return value SSL_ERROR_WANT_ASYNC from any call to SSL_get_error(). See:
https://www.openssl.org/docs/man1.1.1/man3/SSL_get_error.html
Your application can then implement a callback in the normal way via SSL_CTX_set_cert_verify_callback. If it needs to temporarily pause operation to wait for some asynchronous operation to complete then the callback should call ASYNC_pause_job(). See:
https://www.openssl.org/docs/man1.1.1/man3/ASYNC_pause_job.html
This will have the effect of control returning to your main application and SSL_get_error() will return SSL_ERROR_WANT_ASYNC. You will need to implement some mechanism for your application to know when asynchronous processing is complete. When it is, then simply retry the SSL I/O call that was previously paused. Note: this must occur on the same thread that the original call was made from. Your callback will then resume from the point at which it previously paused.