I am confused why "sub_18054DFD0" function use the address of "crypto\rsa\rsa_ossl.c" as a parameter but not using it.(
"crypto\rsa\rsa_ossl.c" is a file in the openssl api library.)
I want to figure out did it call the function in the rsa_ossl library?
.text:00000001805BF4E7 loc_1805BF4E7: ; DATA XREF: .rdata:0000000180902990o
.text:00000001805BF4E7 ; .rdata:00000001809029A0o ...
.text:00000001805BF4E7 mov [rsp+28h+arg_50], r15
.text:00000001805BF4EF call sub_1805AF500 ; STR: "crypto\bn\bn_ctx.c"
.text:00000001805BF4F4 mov rcx, rsi
.text:00000001805BF4F7 call sub_1805AF3D0 ; STR: "crypto\bn\bn_ctx.c"
.text:00000001805BF4FC mov rcx, rsi
.text:00000001805BF4FF mov r15, rax
.text:00000001805BF502 call sub_1805AF3D0 ; STR: "crypto\bn\bn_ctx.c"
.text:00000001805BF507 mov rcx, [rbx+18h]
.text:00000001805BF50B mov rdi, rax
.text:00000001805BF50E call sub_1805614B0
.text:00000001805BF513 add eax, 7
.text:00000001805BF516 mov r8d, 100h
.text:00000001805BF51C cdq
.text:00000001805BF51D and edx, 7
.text:00000001805BF520 add eax, edx
.text:00000001805BF522 lea rdx, aCryptoRsaRsa_o ; "crypto\\rsa\\rsa_ossl.c"
.text:00000001805BF529 sar eax, 3
.text:00000001805BF52C movsxd rbp, eax
.text:00000001805BF52F mov rcx, rbp
.text:00000001805BF532 call sub_18054DFD0
Step in the function "sub_18054DFD0":
.text:000000018054DFD0 sub_18054DFD0 proc near ; CODE XREF: sub_1804FE630+42Ap
.text:000000018054DFD0 ; sub_1804FFAE0+AAp ...
.text:000000018054DFD0 mov eax, 28h
.text:000000018054DFD5 call sub_180670B80
.text:000000018054DFDA sub rsp, rax
.text:000000018054DFDD mov rax, cs:off_18091E640
.text:000000018054DFE4 test rax, rax
.text:000000018054DFE7 jz short loc_18054DFFC
.text:000000018054DFE9 lea r9, sub_18054DFD0
.text:000000018054DFF0 cmp rax, r9
.text:000000018054DFF3 jz short loc_18054DFFC
.text:000000018054DFF5 add rsp, 28h
.text:000000018054DFF9 jmp rax
.text:000000018054DFFC ; ---------------------------------------------------------------------------
.text:000000018054DFFC
.text:000000018054DFFC loc_18054DFFC: ; CODE XREF: sub_18054DFD0+17j
.text:000000018054DFFC ; sub_18054DFD0+23j
.text:000000018054DFFC test rcx, rcx
.text:000000018054DFFF jnz short loc_18054E008
.text:000000018054E001 xor eax, eax
.text:000000018054E003 add rsp, 28h
.text:000000018054E007 retn
.text:000000018054E008 ; ---------------------------------------------------------------------------
.text:000000018054E008
.text:000000018054E008 loc_18054E008: ; CODE XREF: sub_18054DFD0+2Fj
.text:000000018054E008 mov cs:dword_18091E638, 0
.text:000000018054E012 add rsp, 28h
.text:000000018054E016 jmp j_malloc_0
.text:000000018054E016 sub_18054DFD0 endp ; sp-analysis failed
Compiler: MSVC
Architecture：x86-64bit
The file is a mixed DLL and this part is C++ code.
That's an argument for CRYPTO_malloc
As it's easy to see, this function will pass its arguments to malloc_impl if not itself or, otherwise, fallback to malloc.
OpenSSL (and its forks, like BoringSSL) use this indirection to enhance malloc, see this
The string crypto\rsa\rsa_ossl.c is where the function in your first block of code comes from.
Since OpenSSL is opensource and there is source file name information, it's pretty easy to pinpoint the code you are looking at:
It's inside rsa_ossl_private_encrypt that is defined, of course, in crypto\rsa\rsa_ossl.c.
It's a spot on because there are three calls to "BN functions" (a big numbers facility, I guess), two of which identical, and a call to a "memory allocation function".
One may be thrown off by the calculations of the memory to allocate, particularly one may search for the constants but, as the source shows, they are done by an inlined macro BN_num_bytes and thus, searching for those constants (i.e. 256, 7, 8 and all their other numerals) won't give anything.
As you can see, the function at sub_18054DFD0 is OPENSSL_malloc and it takes only one parameter.
OPENSSL_malloc will pass OPENSSL_FILE (which, as this doc pod says, it's just an alias for __FILE__) as the second parameter of CRYPTO_malloc.
It also passes the line number as the third parameter, in your case it's 256 (the value of r8d). My manual analysis found the call a few lines below, which may be due to a different version.
These memory allocations routines keep tracks of the file and line numbers of the allocations callers to ease the debug of memory leaks and corrupts.
Without a tool like Valgrind or DrMemory (for Windows) it's really really hard to pinpoint bug due to memory corruption, so this makes sense.