According to Bouncy Castle FIPS Java API guide, version: 0.1
Under section 4.2, some padding mechanisms are specified to be used with Public Key algorithms (ELGAMAL/RSA):
How one can configure which padding mechanism will be used with public key algorithm? Both for client & server communications.
Specially I am interested in case while RSA used as key exchange mechanism in TLS.
When RSA is used for key transport in TLS plain-RSA keyexchange it always uses PKCS1-v1_5 padding; see rfc5246 4.7 and 7.4.7.1 or the corresponding earlier versions (when it was simply called block type 2, since OAEP didn't exist yet). You can't configure anything else, and anything else wouldn't interoperate. (In TLS1.3 as currently proposed RSA key transport will no longer exist.)
When RSA is used for server or client authentication in TLS, as part of keyexchanges like DHE_RSA and ECDHE_RSA, in 1.0 and 1.1 the signature algorithm is fixed as a modification of PKCS1-v1_5 (using concatenated MD5 and SHA1 hashes and no ASN.1 DigestInfo); see rfc4346 4.7 or rfc2246. In 1.2 the signature algorithm is selected from among those offered by the peer (in a ClientHello extension for the server, in a CertReq field for the client) but the only RSA options are standard PKCS1-v1_5 with MD5 SHA-1 or SHA-2 hashes, and many if not all implementations (including recent updates of Java) now prohibit MD5; AFAICT there is no configuration in JSSE to choose among the permitted hashes. (1.3 proposes to change to PSS in the protocol, with PKCS1-v1_5 signatures allowed only for certificates.)
If you are concerned with FIPS-140 compliance for US government applications, see SP800-135 which Approves the TLS KDFs when used in TLS only although they are not generally Approved for FIPS-140. Also note there is extensive mandatory guidance in SP800-52 for other parts of TLS besides the cryptographic primitives.