RAND_bytes not invoking though setting a RAND_set_rand_method()?

Question

Even though we set currentMethod.bytes with local function to generate random numbers, the RAND_bytes is not invoking. After we set RAND_set_rand_method(&cuurentMethod).

Here I attached link [https://github.com/openssl/openssl/blob/master/test/sm2_internal_test.c] which I already tried.

int main()
{
    unsigned char rand[16];
    int ret;
    RAND_METHOD *oldMethod,currentMethod,*temp;
    oldMethod = RAND_get_rand_method();/*getting default method*/
    currentMethod = *oldMethod;
    currentMethod.bytes = local_function_rand;

    if((ret = RAND_set_rand_method(&currentMethod))!= 1)
        return 0; 

   /* Now we are printing both address of local_function_method_rand() and 
   temp->bytes , those address are same after getting. */

   temp = RAND_get_rand_method();

   /* after we are comparing with RAND_SSLeay() function , to find default or not*/

   if((ret = RAND_bytes(rand,16)) != 1)
       return 0;
   return 1;
}

Expecting result is our local function should invoke. Also, to invoke RAND_bytes() is it required to set fips mode in Linux system?

Answer 1

After cleaning up and minimizing your test program and filling in the missing parts:

#include <openssl/rand.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int local_function_rand(unsigned char *buf, int num) {
  printf("in local_function_rand(); requested %d bytes\n", num);
  memset(buf, 4, num); // RFC 1149.5 standard random number
  return 1;
}

int main(void) {
  unsigned char rand[16];
  RAND_METHOD currentMethod = {.bytes = local_function_rand};
  RAND_set_rand_method(&currentMethod);

  if (RAND_bytes(rand, sizeof rand) != 1) {
    return EXIT_FAILURE;
  }

  return 0;
}

and running it (With OpenSSL 1.1.1):

$ gcc -Wall -Wextra rand.c -lcrypto
$ ./a.out
in local_function_rand(); requested 16 bytes

it works as expected; the user-supplied function is being called by RAND_bytes(). If you're getting different results from your code, there's probably a problem in the bits you didn't include in your question.