I upgraded WSO2 v4.1.0 to v4.2.0 as the deployment.toml is identical, only one config was added in the new version the rest configs are the same.
[apim.key_manager]
enable_apikey_subscription_validation = true
The previous version with all the same configurations started and made requests clearly. With version 4.2.0 the program starts fine, only at the end in wso2carbon.log there are several warnings, I can login to the /carbon page and can open the main devportal page with APIs but as soon as I want to go to the login page on /devportal, /publisher, /admin gives me a 500 status code and the following error in the logs (also attached startup warnings):
TID: [] [] [2023-03-29 17:25:08,220] INFO {org.wso2.carbon.event.processor.core.EventProcessorDeployer} - Execution plan is deployed successfully and in active state : carbon.super_resource_10KPerMin_default
TID: [] [] [2023-03-29 17:25:08,949] WARN {org.wso2.carbon.apimgt.common.jms.JMSListener} - Polling tasks on destination : keyManager of type topic for listener Siddhi-JMS-Consumer#keyManager have not yet started after 3 seconds ..
TID: [] [] [2023-03-29 17:25:08,956] WARN {org.wso2.carbon.apimgt.common.jms.JMSListener} - Polling tasks on destination : notification of type topic for listener Siddhi-JMS-Consumer#notification have not yet started after 3 seconds ..
TID: [] [] [2023-03-29 17:25:09,115] WARN {org.wso2.carbon.apimgt.common.jms.JMSListener} - Polling tasks on destination : notification of type topic for listener Siddhi-JMS-Consumer#notification have not yet started after 3 seconds ..
TID: [] [] [2023-03-29 17:25:09,628] WARN {org.wso2.carbon.apimgt.common.jms.JMSListener} - Polling tasks on destination : tokenRevocation of type topic for listener Siddhi-JMS-Consumer#tokenRevocation have not yet started after 3 seconds ..
TID: [] [] [2023-03-29 17:25:09,690] WARN {org.wso2.carbon.apimgt.common.jms.JMSListener} - Polling tasks on destination : notification of type topic for listener Siddhi-JMS-Consumer#notification have not yet started after 3 seconds ..
TID: [] [] [2023-03-29 17:25:09,728] WARN {org.wso2.carbon.apimgt.common.jms.JMSListener} - Polling tasks on destination : throttleData of type topic for listener Siddhi-JMS-Consumer#throttleData have not yet started after 3 seconds ..
TID: [] [] [2023-03-29 17:25:09,807] WARN {org.wso2.carbon.apimgt.common.jms.JMSListener} - Polling tasks on destination : cacheInvalidation of type topic for listener Siddhi-JMS-Consumer#cacheInvalidation have not yet started after 3 seconds ..
TID: [] [] [2023-03-29 17:25:09,848] WARN {org.wso2.carbon.apimgt.common.jms.JMSListener} - Polling tasks on destination : asyncWebhooksData of type topic for listener Siddhi-JMS-Consumer#asyncWebhooksData have not yet started after 3 seconds ..
TID: [] [] [2023-03-29 17:25:09,849] WARN {org.wso2.carbon.apimgt.common.jms.JMSListener} - Polling tasks on destination : throttleData of type topic for listener Siddhi-JMS-Consumer#throttleData have not yet started after 3 seconds ..
TID: [] [] [2023-03-29 17:26:03,076] WARN {org.apache.synapse.transport.http.access.AccessConfiguration} - Error loading properties from file: access-log.properties
TID: [] [] [2023-03-29 17:26:03,080] WARN {org.apache.synapse.commons.util.MiscellaneousUtil} - Error loading properties from a file at from the System defined location: access-log.properties
TID: [] [] [2023-03-29 17:26:03,088] WARN {org.apache.synapse.commons.util.MiscellaneousUtil} - Error loading properties from a file at from the System defined location: access-log.properties
TID: [] [] [2023-03-29 17:26:03,138] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = /, HEALTH CHECK URL = /
TID: [-1234] [api/am/devportal] [2023-03-29 17:26:07,726] ERROR {org.wso2.carbon.apimgt.rest.api.util.impl.OAuthOpaqueAuthenticatorImpl} - Invalid OAuth Token : Invalid Access Token. ACTIVE access token is not found.
TID: [-1234] [api/am/devportal] [2023-03-29 17:26:07,726] ERROR {org.wso2.carbon.apimgt.rest.api.util.impl.OAuthOpaqueAuthenticatorImpl} - Provided access token is invalid
TID: [-1234] [devportal] [2023-03-29 17:26:14,362] ERROR {org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/devportal].[idp]} - Servlet.service() for servlet [idp] in context with path [/devportal] threw exception java.io.IOException: An exception occurred processing [/services/login/idp.jsp] at line [75]
72: HttpRequest getReq = HttpRequest.newBuilder()
73: .uri(URI.create(settingsAPIUrl))
74: .build();
75: HttpResponse<String> settingsResult = client.send(getReq, HttpResponse.BodyHandlers.ofString());
76:
77: Gson gson = new GsonBuilder().setPrettyPrinting().create();
78: Map settingsResponse = gson.fromJson(settingsResult.body(), Map.class);
Stacktrace:
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:494)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.wso2.carbon.ui.filters.cache.ContentTypeBasedCachePreventionFilter.doFilter(ContentTypeBasedCachePreventionFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:126)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:119)
at org.wso2.carbon.identity.context.rewrite.valve.OrganizationContextRewriteValve.invoke(OrganizationContextRewriteValve.java:116)
at org.wso2.carbon.tomcat.ext.valves.SameSiteCookieValve.invoke(SameSiteCookieValve.java:38)
at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:165)
at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:111)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:106)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:49)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:67)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:152)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:63)
at org.wso2.carbon.tomcat.ext.valves.RequestCorrelationIdValve.invoke(RequestCorrelationIdValve.java:137)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLHandshakeException: No name matching localhost found
at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:578)
at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:123)
at org.apache.jsp.services.login.idp_jsp._jspService(idp_jsp.java:206)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:466)
... 42 more
Caused by: javax.net.ssl.SSLHandshakeException: No name matching localhost found
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate.lambda$executeTasks$3(SSLFlowDelegate.java:1118)
at java.net.http/jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:157)
at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate.executeTasks(SSLFlowDelegate.java:1113)
at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate.doHandshake(SSLFlowDelegate.java:1079)
at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:484)
at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:268)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:205)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
... 1 more
Caused by: java.security.cert.CertificateException: No name matching localhost found
at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:234)
at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:458)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:418)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
... 21 more
Running on Centos7 with Java17.
I am also attaching deployment.toml with configs which work on version 4.1.0, the same config I use for 4.2.0:
[server]
hostname = "{{ hostname }}"
#offset=0
base_path = "${carbon.protocol}://${carbon.host}:${carbon.management.port}"
#discard_empty_caches = false
server_role = "default"
[server.file_upload]
file_size_limit = "0"
[transport.https.properties]
proxyPort = 443
[super_admin]
username = "{{ apigw_server.admin.username }}"
password = "$secret{admin_password}"
create_admin_account = true
[user_store]
type = "database_unique_id"
[database.apim_db]
type = "mysql"
url = "jdbc:mysql://localhost:3306/{{ db.schemas.apim.apim_db }}"
username = "{{ db.user.username }}"
password = "$secret{wso2am_db_password}"
driver = "com.mysql.cj.jdbc.Driver"
[database.shared_db]
type = "mysql"
url = "jdbc:mysql://localhost:3306/{{ db.schemas.apim.shared_db }}"
username = "{{ db.user.username }}"
password = "$secret{wso2am_db_password}"
driver = "com.mysql.cj.jdbc.Driver"
#[keystore.tls]
#file_name = "wso2carbon.jks"
#type = "JKS"
#password = "wso2carbon"
#alias = "wso2carbon"
#key_password = "wso2carbon"
#[keystore.listener_profile]
#bind_address = "0.0.0.0"
[keystore.primary]
file_name = "wso2carbon.jks"
type = "JKS"
password = "$secret{keystore_password}"
alias = "wso2carbon"
key_password = "$secret{keystore_key_password}"
[keystore.internal]
file_name = "wso2carbon.jks"
type = "JKS"
password = "$secret{keystore_password}"
alias = "wso2carbon"
key_password = "$secret{keystore_key_password}"
[[apim.gateway.environment]]
name = "Default"
type = "hybrid"
provider = "wso2"
display_in_api_console = true
description = "This is a hybrid gateway that handles both production and sandbox token traffic."
show_as_token_endpoint_url = true
service_url = "https://{{ hostname }}:${mgt.transport.https.port}/services/"
username= "${admin.username}"
password= "${admin.password}"
ws_endpoint = "ws://{{ hostname }}"
wss_endpoint = "wss://{{ hostname }}"
http_endpoint = "http://{{ hostname }}"
https_endpoint = "https://{{ hostname }}"
websub_event_receiver_http_endpoint = "http://{{ hostname }}:9021"
websub_event_receiver_https_endpoint = "https://{{ hostname }}:8021"
[apim.sync_runtime_artifacts.gateway]
gateway_labels =["Default"]
#[apim.cache.gateway_token]
#enable = true
#expiry_time = "900s"
[apim.cache.resource]
enable = false
#expiry_time = "900s"
#[apim.cache.km_token]
#enable = false
#expiry_time = "15m"
#[apim.cache.recent_apis]
#enable = false
#[apim.cache.scopes]
#enable = true
#[apim.cache.publisher_roles]
#enable = true
[apim.cache.jwt_claim]
enable = false
#expiry_time = "15m"
#[apim.cache.tags]
#expiry_time = "2m"
[apim.analytics]
enable = false
auth_token = ""
[apim.key_manager]
enable_apikey_subscription_validation = true
#service_url = "https://{{ hostname }}/services/"
#username = "$ref{super_admin.username}"
#password = "$ref{super_admin.password}"
#pool.init_idle_capacity = 50
#pool.max_idle = 100
#key_validation_handler_type = "default"
#key_validation_handler_type = "custom"
#key_validation_handler_impl = "org.wso2.carbon.apimgt.keymgt.handlers.DefaultKeyValidationHandler"
#[apim.idp]
#server_url = "https://{{ hostname }}"
#authorize_endpoint = "https://{{ hostname }}/oauth2/authorize"
#oidc_logout_endpoint = "https://{{ hostname }}/oidc/logout"
#oidc_check_session_endpoint = "https://{{ hostname }}/oidc/checksession"
[apim.jwt]
enable = true
encoding = "base64" # base64,base64url
#generator_impl = "org.wso2.carbon.apimgt.keymgt.token.JWTGenerator"
claim_dialect = "https://some_domain.com/claims"
convert_dialect = true
header = "X-JWT-Assertion"
signing_algorithm = "NONE"
enable_user_claims = true
claims_extractor_impl = "org.wso2.carbon.apimgt.impl.token.DefaultClaimsRetriever"
[apim.jwt.gateway_generator]
impl = "com.some_domain.api.gateway.JwtTokenGenerator"
#[apim.oauth_config]
#enable_outbound_auth_header = false
#auth_header = "Authorization"
#revoke_endpoint = "https://some_domain.com:${https.nio.port}/revoke"
#enable_token_encryption = false
#enable_token_hashing = false
[apim.devportal]
url = "https://{{ hostname }}/devportal"
display_url = true
#enable_application_sharing = false
#if application_sharing_type, application_sharing_impl both defined priority goes to application_sharing_impl
#application_sharing_type = "default" #changed type, saml, default #todo: check the new config for rest api
#application_sharing_impl = "org.wso2.carbon.apimgt.impl.SAMLGroupIDExtractorImpl"
#display_multiple_versions = false
#display_deprecated_apis = false
#enable_comments = true
#enable_ratings = true
#enable_forum = true
#enable_anonymous_mode=true
#enable_cross_tenant_subscriptions = true
#default_reserved_username = "apim_reserved_user"
[apim.cors]
allow_origins = "*"
allow_methods = ["GET","PUT","POST","DELETE","PATCH","OPTIONS"]
allow_headers = [{{ cors.allowed_headers }}]
allow_credentials = false
#[apim.throttling]
#enable_data_publishing = true
#enable_policy_deploy = true
#enable_blacklist_condition = true
#enable_persistence = true
#throttle_decision_endpoints = ["tcp://localhost:5672","tcp://localhost:5672"]
#[apim.throttling.blacklist_condition]
#start_delay = "5m"
#period = "1h"
#[apim.throttling.jms]
#start_delay = "5m"
#[apim.throttling.event_sync]
#hostName = "0.0.0.0"
#port = 11224
#[apim.throttling.event_management]
#hostName = "0.0.0.0"
#port = 10005
#[[apim.throttling.url_group]]
#traffic_manager_urls = ["tcp://localhost:9611","tcp://localhost:9611"]
#traffic_manager_auth_urls = ["ssl://localhost:9711","ssl://localhost:9711"]
#type = "loadbalance"
#[[apim.throttling.url_group]]
#traffic_manager_urls = ["tcp://localhost:9611","tcp://localhost:9611"]
#traffic_manager_auth_urls = ["ssl://localhost:9711","ssl://localhost:9711"]
#type = "failover"
#[apim.workflow]
#enable = false
#service_url = "https://localhost:9445/bpmn"
#username = "$ref{super_admin.username}"
#password = "$ref{super_admin.password}"
#callback_endpoint = "https://localhost:${mgt.transport.https.port}/api/am/admin/v0.17/workflows/update-workflow-status"
#token_endpoint = "https://localhost:${https.nio.port}/token"
#client_registration_endpoint = "https://localhost:${mgt.transport.https.port}/client-registration/v0.17/register"
#client_registration_username = "$ref{super_admin.username}"
#client_registration_password = "$ref{super_admin.password}"
#data bridge config
#[transport.receiver]
#type = "binary"
#worker_threads = 10
#session_timeout = "30m"
#keystore.file_name = "$ref{keystore.tls.file_name}"
#keystore.password = "$ref{keystore.tls.password}"
#tcp_port = 9611
#ssl_port = 9711
#ssl_receiver_thread_pool_size = 100
#tcp_receiver_thread_pool_size = 100
#ssl_enabled_protocols = ["TLSv1","TLSv1.1","TLSv1.2"]
#ciphers = ["SSL_RSA_WITH_RC4_128_MD5","SSL_RSA_WITH_RC4_128_SHA"]
#[apim.notification]
#from_address = "APIM.com"
#username = "APIM"
#password = "APIM+123"
#hostname = "localhost"
#port = 3025
#enable_start_tls = false
#enable_authentication = true
#[apim.token.revocation]
#notifier_impl = "org.wso2.carbon.apimgt.keymgt.events.TokenRevocationNotifierImpl"
#enable_realtime_notifier = true
#realtime_notifier.ttl = 5000
#enable_persistent_notifier = true
#persistent_notifier.hostname = "https://localhost:2379/v2/keys/jti/"
#persistent_notifier.ttl = 5000
#persistent_notifier.username = "root"
#persistent_notifier.password = "root"
[[event_handler]]
name="userPostSelfRegistration"
subscriptions=["POST_ADD_USER"]
[service_provider]
sp_name_regex = "^[\\sa-zA-Z0-9._-]*$"
[database.local]
type = "mysql"
url = "jdbc:mysql://localhost:3306/{{ db.schemas.apim.local_db }}"
username = "{{ db.user.username }}"
password = "{{ db.user.password }}"
driver = "com.mysql.cj.jdbc.Driver"
[[event_listener]]
id = "token_revocation"
type = "org.wso2.carbon.identity.core.handler.AbstractIdentityHandler"
name = "org.wso2.is.notification.ApimOauthEventInterceptor"
order = 1
[event_listener.properties]
notification_endpoint = "https://{{ hostname }}/internal/data/v1/notify"
username = "${admin.username}"
password = "${admin.password}"
'header.X-WSO2-KEY-MANAGER' = "default"
[oauth.grant_type.token_exchange]
enable = true
allow_refresh_tokens = true
iat_validity_period = "1h"
[oauth.token_validation]
refresh_token_validity = 84600
How to fix it?
The error indicates that it's an SSL HostName verification issue. So I assume the certificate you have configured doesn't have localhost as the Common Name(CN) or as a Subject Alternate Name(SAN) hence when the server is trying to connect with the host localhost the HostName verification fails. There are different ways to resolve this, take a look at this answer.
As a temporary solution can you try starting API Manager with the following flag?
sh api-manager.sh -Djdk.internal.httpclient.disableHostnameVerification=true
If the above doesn't yield any results, try following two properties as well. You can pass all 3 together.
-Dorg.opensaml.httpclient.https.disableHostnameVerification=true
-Dhttpclient.hostnameVerifier=AllowAll