Is there any way through which a user process can have dynamic memory allocation, without using system call? I know malloc uses brk() or mmap() system calls to allocate memory.
Can we do dynamic memory allocation, without system calls?
In practice, for most (but not all) operating systems, yes.
Specifically, if you reserve a huge area in your ".bss" section, then most operating systems won't allocate physical RAM for that area until its written to, and you can implement some kind of heap ("malloc()") that uses the reserved huge area.
Note: while this can avoid switching to/from kernel for system calls, it won't avoid switching to/from kernel for page faults.
I know malloc uses brk() or mmap() system calls to allocate memory.
That's a "language run-time specific" thing. There's no reason (for a language like C) the standard library can't do the same "huge reserved section" trick; and no reason you can't write a program that doesn't use "malloc()" at all (the program I'm currently writing in C does this because "malloc()" is a nasty piece of brokenness), and no reason for a different language (e.g. maybe something stack-based like LISP?) to have anything like the horrible POSIX/C mess.