Our server can support both async and sync HTTP requests. However processing of async request depends on a number of internal parameter and "can-not-be-supported" in some instance.
Client requests for async processing of request using prefer header.
https://www.rfc-editor.org/rfc/rfc7240#page-8
What shall be the proper response code by server if async is not supported? Shall it be HTTP 501 not implemented?
https://www.rfc-editor.org/rfc/rfc7231#section-6.6.2
Note that same request can be processed synchronously if customer does not include prefer header. So can this be also treated as 4xx error?
Shall it be HTTP 501 not implemented?
No.
So can this be also treated as 4xx error?
Not at all.
What shall be the proper response code by server if async is not supported?
Ignore it and move on. See what the RFC 7240 says:
A server that does not recognize or is unable to comply with particular preference tokens in the
Preferheader field of a request MUST ignore those tokens and continue processing instead of signaling an error.