I need to monitor the Oracle Cloud Compute VM using REST API's or Java SDK. So for REST API monitoring do we need to do any set up in VM?
No, you don't need any set up inside the VM, but you do need the API users to be authorized in a IAM policy.
For example with a policy like this (for just reading the metrics):
Allow group <api_users> to read metrics in tenancy
Then you can use the APIs here.