we need to expose an internal API to vendor, however for development we need to mask the data the internal API replies, looking for a similar feature with apigee (https://docs.apigee.com/api-platform/security/data-masking), can this doable via APIM?
Generically, the Azure APIM allows flexibility on the handling of requests and responses, mainly with the concept of policies. In short, these are operations that may be defined in the incoming, outgoing or during the execution of requests.
This allows the changing of the requests/responses (for example, adding or removing headers), which may be what you want. Some typical examples/snippets of policies are depicted here in Azure's own GitHub, as well as in APIM itself, as shown here.
These two walkthroughs may also be helpful: Setting Policies and Transforming APIs.