Not able to access Publisher URL after SSL certificate renewal

We have renewed our self-signed internal certificate in WSO2 API Manager 3.2 as below and after that we are not able to access the Publisher and Dev Portal URLs. However we are able to access the Carbon URL.

Here are the steps followed.

keytool -genkey -alias wso2mcm -keyalg RSA -keysize 2048 -keystore wso2mcm.jks -dname "CN=ad.local.com, OU=Home,O=Home,L=SL,S=WS,C=LK" --storepass wso2carbon -keypass wso2carbon -validity 365
keytool -export -alias wso2mcm -keystore wso2mcm.jks -file wso2mcm.pem
keytool -delete -alias wso2mcm -keystore client-truststore.jks
keytool -import -alias wso2mcm -file wso2mcm.pem -keystore client-truststore.jks -storepass wso2carbon
keytool -importkeystore -srckeystore wso2mcm.jks -destkeystore wso2mcm.p12 -deststoretype PKCS12

Entries from deployment.toml file are as below

[keystore.tls]
file_name =  "wso2mcm.jks"
type =  "JKS"
password =  "wso2carbon"
alias =  "wso2mcm"
key_password =  "wso2carbon"

[keystore.primary]
file_name =  "wso2carbon.jks"
type =  "JKS"
password =  "wso2carbon"
alias =  "wso2carbon"
key_password =  "wso2carbon"

[keystore.internal]
file_name =  "wso2carbon.jks"
type =  "JKS"
password =  "wso2carbon"
alias =  "wso2carbon"
key_password =  "wso2carbon"

Is there any additional steps to be followed? Please help us to resolve this issue

Please find below the error screenshot and log entries from carbon log file

TID: [-1234] [] [2022-12-16 13:10:11,367] ERROR {org.wso2.carbon.identity.oauth2.OAuth2Service} - Error occurred while issuing the access token for Client ID : P_3DbvpsLEMzJJa89HsBFEjEJaUa, User ID null, Scope : [] and Grant Type : authorization_code java.lang.NullPointerException
        at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
        at java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
        at org.wso2.carbon.identity.oauth2.util.OAuth2Util.getPrivateKey(OAuth2Util.java:2370)
        at org.wso2.carbon.identity.oauth2.util.OAuth2Util.signJWTWithRSA(OAuth2Util.java:2327)
        at org.wso2.carbon.identity.oauth2.util.OAuth2Util.signJWT(OAuth2Util.java:2287)
        at org.wso2.carbon.identity.openidconnect.DefaultIDTokenBuilder.getIDToken(DefaultIDTokenBuilder.java:295)
        at org.wso2.carbon.identity.openidconnect.DefaultIDTokenBuilder.buildIDToken(DefaultIDTokenBuilder.java:204)
        at org.wso2.carbon.identity.oauth2.token.AccessTokenIssuer.issue(AccessTokenIssuer.java:338)
        at org.wso2.carbon.identity.oauth2.OAuth2Service.issueAccessToken(OAuth2Service.java:254)
        at org.wso2.carbon.identity.oauth.endpoint.token.OAuth2TokenEndpoint.issueAccessToken(OAuth2TokenEndpoint.java:318)
        at org.wso2.carbon.identity.oauth.endpoint.token.OAuth2TokenEndpoint.issueAccessToken(OAuth2TokenEndpoint.java:94)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179)
        at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:193)
        at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:103)
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
        at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267)
        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:216)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:220)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:276)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.wso2.carbon.webapp.mgt.filter.AuthorizationHeaderFilter.doFilter(AuthorizationHeaderFilter.java:128)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.wso2.carbon.ui.filters.cache.ContentTypeBasedCachePreventionFilter.doFilter(ContentTypeBasedCachePreventionFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:86)
        at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:110)
        at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:101)
        at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
        at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:49)
        at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
        at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:145)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
        at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
        at org.wso2.carbon.tomcat.ext.valves.RequestCorrelationIdValve.invoke(RequestCorrelationIdValve.java:119)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Solution

I followed the same commands you mentioned and was able to successfully update the certificate.

The error you received occurs when the primary keystore password and key password are different. During the certificate update, you might have changed the key password of wso2carbon.jks. Can you try the below command and change it back to wso2carbon?

keytool -keypasswd -keystore wso2carbon.jks -alias wso2carbon

This will ask for the existing password for keystore and the private key. If wso2carbon does not match, that means password has been changed and either you have to use the correct password or create a new keystore and use it as primary and internal keystore. For your testing purposes, you can even replace the Keystore from another APIM distribution.

Update

Since you have changed the client secret of the publisher SP app, you either have to update the SP app clientSecret in the AM_SYSTEM_APPS table or create a new SP for the publisher login.

You can create a new SP with,

Delete the existing SP app with the name apim_publisher
Delete the apim_publisher entry from AM_SYSTEM_APPS table.
Login again from the publisher portal.