WSO2 Developer Portal API: map-keys endpoint does not work

I'm migrating from WSO2 2.6 over to WSO2 4.1. So far faced with an issue that I can not import consumerKey and consumerSecret using DevPortl API. My import flow looks like this:

Application created thru API in the following way (where {{basepath}} is https://{host}/api/am/devportal/v2.1):

Endpoint: POST {{basepath}}/applications

Request:

{
  "name": "Testing import",
  "throttlingPolicy": "Unlimited",
  "description": "Test description",
  "tokenType": "JWT"
}

Response:

{
    "applicationId": "79d77586-4f5f-4ea6-8260-2c59188e423c",
    "name": "Testing import",
    "throttlingPolicy": "Unlimited",
    "description": "Test description",
    "tokenType": "JWT",
    "status": "APPROVED",
    "groups": [],
    "subscriptionCount": 0,
    "keys": [],
    "attributes": {},
    "subscriptionScopes": [],
    "owner": "CARCTEMY/apiuser",
    "hashEnabled": null
}

After that for the newly created application I'm invoking POST {{basepath}}/applications/79d77586-4f5f-4ea6-8260-2c59188e423c/map-keys with consumerKey and consumerSecret created in WSO2 2.6 version:

{
  "consumerKey": "KBs51iQITZK02v5oPwSRewK7q_Qa",
  "consumerSecret": "Qt2u7INdReROhpPI8nbedyDBOIYa",
  "keyManager": "Resident Key Manager",
  "keyType": "PRODUCTION"
}

and as a response I'm getting following error:

{
    "code": 900967,
    "message": "General Error",
    "description": "Server Error Occurred",
    "moreInfo": "",
    "error": []
}

Logs are attached below.

As far as I understand from logs KeyManager requires some additional authentication steps, but at the same time I can generate key and secret thru POST {{basepath}}/applications/{applicationId}/generate-keys endpoint, thus that 401 error confuses me. Also, if I first run /generate-keys and then /map-keys the second request returns me an error saying "Key Mappings already exists"

Am I missing some additional configs for KeyManager?

UPD

I've also tried to call /map-keys endpoint as a WSO admin user with consumerKey and consumerSecret generated by another instance of WSO2 v4.1 but got the same error. Bearer token generated with following scopes: apim:admin apim:api_key apim:app_import_export apim:app_manage apim:store_settings apim:sub_alert_manage apim:sub_manage apim:subscribe apim:subscribe apim:api_view apim:api_create apim:api_publish apim:tier_view

However, at the same time /generate-keys endpoint work. So I assume that /map-keys is broken or requires some additional scopes I'm not aware of.

http_acces.log:

127.0.0.1 - - [13/Feb/2023:18:35:10 +0200] POST /api/am/devportal/v2.1/applications/79d77586-4f5f-4ea6-8260-2c59188e423c/map-keys HTTP/1.1 500 104 - PostmanRuntime/7.30.1 0.029
127.0.0.1 - - [13/Feb/2023:18:35:10 +0200] GET /keymanager-operations/dcr/register/S0JzNTFpUUlUWkswMnY1b1B3U1Jld0s3cV9RYQ%3D%3D HTTP/1.1 401 - - Apache-HttpClient/4.5.13 (Java/11.0.18) 0.009

wso2-apigw-errors.log:

2023-02-13T18:35:10,942 [-] [https-jsse-nio-9443-exec-21] ERROR AbstractKeyManager Some thing went wrong while getting OAuth application for given consumer key KBs51iQITZK02v5oPwSRewK7q_Qa
org.wso2.carbon.apimgt.impl.kmclient.KeyManagerClientException: Received status code: 401 Reason: 
    at org.wso2.carbon.apimgt.impl.kmclient.KMClientErrorDecoder.decode_aroundBody0(KMClientErrorDecoder.java:42) ~[org.wso2.carbon.apimgt.impl_9.20.74.jar:?]
    at org.wso2.carbon.apimgt.impl.kmclient.KMClientErrorDecoder.decode(KMClientErrorDecoder.java:35) ~[org.wso2.carbon.apimgt.impl_9.20.74.jar:?]
    at feign.AsyncResponseHandler.handleResponse(AsyncResponseHandler.java:96) ~[io.github.openfeign.feign-core_11.0.0.jar:?]
    at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:138) ~[io.github.openfeign.feign-core_11.0.0.jar:?]
    at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89) ~[io.github.openfeign.feign-core_11.0.0.jar:?]
    at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100) ~[io.github.openfeign.feign-core_11.0.0.jar:?]
    at com.sun.proxy.$Proxy467.getApplication(Unknown Source) ~[?:?]
    at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.mapOAuthApplication_aroundBody20(AMDefaultKeyManagerImpl.java:581) ~[org.wso2.carbon.apimgt.impl_9.20.74.jar:?]
    at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.mapOAuthApplication(AMDefaultKeyManagerImpl.java:561) ~[org.wso2.carbon.apimgt.impl_9.20.74.jar:?]
    at org.wso2.carbon.apimgt.impl.APIConsumerImpl.mapExistingOAuthClient_aroundBody78(APIConsumerImpl.java:2517) ~[org.wso2.carbon.apimgt.impl_9.20.74.jar:?]
    at org.wso2.carbon.apimgt.impl.APIConsumerImpl.mapExistingOAuthClient(APIConsumerImpl.java:2452) ~[org.wso2.carbon.apimgt.impl_9.20.74.jar:?]
    at org.wso2.carbon.apimgt.rest.api.store.v1.impl.ApplicationsApiServiceImpl.applicationsApplicationIdMapKeysPost(ApplicationsApiServiceImpl.java:1101) ~[?:?]
    at org.wso2.carbon.apimgt.rest.api.store.v1.ApplicationsApi.applicationsApplicationIdMapKeysPost(ApplicationsApi.java:281) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
    at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) ~[?:?]
    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) ~[?:?]
    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) ~[?:?]
    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) ~[?:?]
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) ~[?:?]
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) ~[?:?]
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) ~[?:?]
    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) ~[?:?]
    at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) ~[?:?]
    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) ~[?:?]
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) ~[?:?]
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) ~[?:?]
    at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) ~[?:?]
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:304) ~[?:?]
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217) ~[?:?]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) ~[tomcat-servlet-api_9.0.58.wso2v1.jar:?]
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:279) ~[?:?]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:107) ~[org.wso2.carbon.identity.context.rewrite.valve_1.4.52.jar:?]
    at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:110) ~[org.wso2.carbon.identity.authz.valve_1.4.52.jar:?]
    at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:102) ~[org.wso2.carbon.identity.auth.valve_1.4.52.jar:?]
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:101) ~[org.wso2.carbon.tomcat.ext_4.6.3.jar:?]
    at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:49) ~[org.wso2.carbon.tomcat.ext_4.6.3.jar:?]
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62) ~[org.wso2.carbon.tomcat.ext_4.6.3.jar:?]
    at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:146) ~[org.wso2.carbon.tomcat.ext_4.6.3.jar:?]
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:58) ~[org.wso2.carbon.tomcat.ext_4.6.3.jar:?]
    at org.wso2.carbon.tomcat.ext.valves.RequestCorrelationIdValve.invoke(RequestCorrelationIdValve.java:126) ~[org.wso2.carbon.tomcat.ext_4.6.3.jar:?]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1735) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat_9.0.58.wso2v1.jar:?]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat_9.0.58.wso2v1.jar:?]
    at java.lang.Thread.run(Thread.java:829) [?:?]
2023-02-13T18:35:10,942 [-] [https-jsse-nio-9443-exec-21] ERROR GlobalThrowableMapper Some thing went wrong while getting OAuth application for given consumer key KBs51iQITZK02v5oPwSRewK7q_Qa

wso2carbon.log:

TID: [-1234] [api/am/devportal] [2023-02-13 18:35:10,942] ERROR {org.wso2.carbon.apimgt.impl.AbstractKeyManager} - Some thing went wrong while getting OAuth application for given consumer key KBs51iQITZK02v5oPwSRewK7q_Qa org.wso2.carbon.apimgt.impl.kmclient.KeyManagerClientException: Received status code: 401 Reason: 
    at org.wso2.carbon.apimgt.impl.kmclient.KMClientErrorDecoder.decode_aroundBody0(KMClientErrorDecoder.java:42)
    at org.wso2.carbon.apimgt.impl.kmclient.KMClientErrorDecoder.decode(KMClientErrorDecoder.java:35)
    at feign.AsyncResponseHandler.handleResponse(AsyncResponseHandler.java:96)
    at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:138)
    at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89)
    at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100)
    at com.sun.proxy.$Proxy467.getApplication(Unknown Source)
    at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.mapOAuthApplication_aroundBody20(AMDefaultKeyManagerImpl.java:581)
    at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.mapOAuthApplication(AMDefaultKeyManagerImpl.java:561)
    at org.wso2.carbon.apimgt.impl.APIConsumerImpl.mapExistingOAuthClient_aroundBody78(APIConsumerImpl.java:2517)
    at org.wso2.carbon.apimgt.impl.APIConsumerImpl.mapExistingOAuthClient(APIConsumerImpl.java:2452)
    at org.wso2.carbon.apimgt.rest.api.store.v1.impl.ApplicationsApiServiceImpl.applicationsApplicationIdMapKeysPost(ApplicationsApiServiceImpl.java:1101)
    at org.wso2.carbon.apimgt.rest.api.store.v1.ApplicationsApi.applicationsApplicationIdMapKeysPost(ApplicationsApi.java:281)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
    at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201)
    at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104)
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
    at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
    at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
    at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265)
    at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
    at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
    at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:304)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
    at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:279)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:107)
    at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:110)
    at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:102)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:101)
    at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:49)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
    at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:146)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
    at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:58)
    at org.wso2.carbon.tomcat.ext.valves.RequestCorrelationIdValve.invoke(RequestCorrelationIdValve.java:126)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1735)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:829)

TID: [-1234] [api/am/devportal] [2023-02-13 18:35:10,942] ERROR {org.wso2.carbon.apimgt.rest.api.util.exception.GlobalThrowableMapper} - Some thing went wrong while getting OAuth application for given consumer key KBs51iQITZK02v5oPwSRewK7q_Qa

Solution

Usually application migration is covered under the migration procedure and seems like you have missed it. However you can still get this done.

Instead of this path, shall we try using the apictl project? APICTL provides the exporting and importing APIs/Applications without much hussle and there is an easy way to achieve your requirement.

Download the APICTL for APIM 4.1.0.
Create an environment with apictl pointing to your deployment, login and list the application. You can find the APICTL commands with this doc
Export a created application which has consumer key, and secret generated. (apictl export app -e dev -n <application_name> -o <app_owner> -k --with-keys)
Extract the downloaded zip file and check the application.yaml
Export the application that doesn't have consumer key,secret generated.
Fill the consumer key and secret section of that applications' application.yaml. (Since this application doesn't have keys exported, you might have to copy the section from the previous exported application. That's why i asked you to export it in the 3rd step for reference)
Here consumer secret is base64 encoded. Therefore you need to encode it and add it here.
Import the updated application with apictl import app -f dev/admin-TestAPP -e dev -k --update. The location of the directory is given with -f flag.
This should map the consumer key secret pair to this application. (If this update does not work, you can delete the application from devportal and import this again. this will create a new application with provided consumer key secret combination).

Let me know whether this worked.