Search code examples
javajenkinsjenkins-plugins

Windows Jenkins Server Unable To Reach Plugin Center Behind Proxy


Summary

Currently, I am trying to install plugins on a windows Jenkins server running behind a proxy. As of now although, the proxy settings are set I am receiving a 403 error when I click the "Check now" button at the bottom of the Advanced tab under plugin manager.

What I tried

I attempted several fixes.

  1. Added the proxy settings in the Plugin Manager Advanced tab. With this included the "validate proxy" button returned an error. I couldn't make out anything useful from the stack trace (Stack trace listed below)
  2. Added the proxy settings in the jenkins.xml file and restarted the Jenkins service. (Arguments added to the Jenkinsfile listed below)
  3. I rebooted the server and validated that I am able to reach the same update center URL through the internet browser and via powershell. However, jenkin's was still getting a 403 error even though these tests were ran from the same machine. (Powershell command used to validate connectivity listed below)

Stack trace from Plugin Manager>Advanced>Validate Proxy against https://www.google.com/

jenkins plugin proxy settings

java.lang.NullPointerException
    at hudson.util.Secret$1.convert(Secret.java:306)
    at hudson.util.Secret$1.convert(Secret.java:304)
    at org.kohsuke.stapler.AnnotationHandler.convert(AnnotationHandler.java:66)
    at org.kohsuke.stapler.QueryParameter$HandlerImpl.parse(QueryParameter.java:74)
    at org.kohsuke.stapler.QueryParameter$HandlerImpl.parse(QueryParameter.java:62)
    at org.kohsuke.stapler.AnnotationHandler.handle(AnnotationHandler.java:91)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:204)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
    at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
    at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
    at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:500)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
    at java.lang.Thread.run(Unknown Source)

Values added to the arguments section of the jenkins.xml file

-Djdk.http.auth.tunneling.disabledSchemes="" -Dhttp.proxyUser=EXAMPLE\user -Dhttp.proxyPassword=EXAMPLEPASS -Dhttp.proxyHost=proxy.example.com -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.example.com -Dhttps.proxyPort=8080 -Dhttp.auth.ntlm.domain=true -Dcom.sun.net.ssl.checkRevocation=false

Powershell command used to successfully validate internet connectivity

Invoke-WebRequest -Uri https://www.google.com `
                  -Proxy http://proxy.example.com:8080 `
                  -ProxyCredential $(Get-Credential)

jenkins.err Log File

Too add some more context to the 403 error. 403 as far as I'm aware suggests it is forbidden, however I do not get any 403 errors if I attempt to reach the site directly through the browser or powershell. If I purposely enter the wrong proxy credentials I get a 407, but I'm able to reach the update center site, https://updates.jenkins.io/update-center.json through the browser and powershell with no errors returned.

2020-04-23 01:54:27.133+0000 [id=1] INFO    org.eclipse.jetty.util.log.Log#initialized: Logging initialized @578ms to org.eclipse.jetty.util.log.JavaUtilLog
2020-04-23 01:54:27.242+0000 [id=1] INFO    winstone.Logger#logInternal: Beginning extraction from war file
2020-04-23 01:54:27.273+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
2020-04-23 01:54:27.336+0000 [id=1] INFO    org.eclipse.jetty.server.Server#doStart: jetty-9.4.27.v20200227; built: 2020-02-27T18:37:21.340Z; git: a304fd9f351f337e7c0e2a7c28878dd536149c6c; jvm 1.8.0_144-b01
2020-04-23 01:54:27.820+0000 [id=1] INFO    o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2020-04-23 01:54:27.883+0000 [id=1] INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
2020-04-23 01:54:27.883+0000 [id=1] INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
2020-04-23 01:54:27.883+0000 [id=1] INFO    o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms
2020-04-23 01:54:28.273+0000 [id=1] INFO    hudson.WebAppMain#contextInitialized: Jenkins home directory: C:\Program Files (x86)\Jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
2020-04-23 01:54:28.367+0000 [id=1] INFO    o.e.j.s.handler.ContextHandler#doStart: Started w.@2f59fa{Jenkins v2.222.1,/,file:///C:/Program%20Files%20(x86)/Jenkins/war/,AVAILABLE}{C:\Program Files (x86)\Jenkins\war}
2020-04-23 01:54:28.383+0000 [id=1] INFO    o.e.j.server.AbstractConnector#doStart: Started ServerConnector@bc464{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2020-04-23 01:54:28.383+0000 [id=1] INFO    org.eclipse.jetty.server.Server#doStart: Started @1827ms
2020-04-23 01:54:28.383+0000 [id=19]    INFO    winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
2020-04-23 01:54:30.617+0000 [id=26]    INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization
2020-04-23 01:54:30.648+0000 [id=25]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
2020-04-23 01:54:31.851+0000 [id=27]    INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2020-04-23 01:54:31.851+0000 [id=27]    INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins
2020-04-23 01:54:31.867+0000 [id=26]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2020-04-23 01:54:32.070+0000 [id=26]    INFO    jenkins.InitReactorRunner$1#onAttained: System config loaded
2020-04-23 01:54:32.070+0000 [id=26]    INFO    jenkins.InitReactorRunner$1#onAttained: System config adapted
2020-04-23 01:54:32.070+0000 [id=26]    INFO    jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
2020-04-23 01:54:32.070+0000 [id=24]    INFO    jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
2020-04-23 01:54:32.086+0000 [id=40]    INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Download metadata
2020-04-23 01:54:32.101+0000 [id=40]    INFO    hudson.util.Retrier#start: Attempt #1 to do the action check updates server
2020-04-23 01:54:32.414+0000 [id=40]    INFO    hudson.util.Retrier#start: The attempt #1 to do the action check updates server failed with an allowed exception:
java.io.IOException: Server returned HTTP response code: 403 for URL: https://updates.jenkins.io/update-center.json?id=default&version=2.222.1
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
    at hudson.model.DownloadService.loadJSON(DownloadService.java:114)
    at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:208)
    at hudson.model.UpdateSite.updateDirectlyNow(UpdateSite.java:203)
    at hudson.PluginManager.checkUpdatesServer(PluginManager.java:1765)
    at hudson.util.Retrier.start(Retrier.java:63)
    at hudson.PluginManager.doCheckUpdatesServer(PluginManager.java:1736)
    at jenkins.DailyCheck.execute(DailyCheck.java:93)
    at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:100)
    at java.lang.Thread.run(Unknown Source)
2020-04-23 01:54:32.414+0000 [id=40]    INFO    hudson.util.Retrier#start: Calling the listener of the allowed exception 'Server returned HTTP response code: 403 for URL: https://updates.jenkins.io/update-center.json?id=default&version=2.222.1' at the attempt #1 to do the action check updates server
2020-04-23 01:54:32.414+0000 [id=40]    INFO    hudson.util.Retrier#start: Attempted the action check updates server for 1 time(s) with no success
2020-04-23 01:54:32.414+0000 [id=40]    SEVERE  hudson.PluginManager#doCheckUpdatesServer: Error checking update sites for 1 attempt(s). Last exception was: IOException: Server returned HTTP response code: 403 for URL: https://updates.jenkins.io/update-center.json?id=default&version=2.222.1
2020-04-23 01:54:32.414+0000 [id=40]    INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 328 ms
2020-04-23 01:54:32.430+0000 [id=27]    INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
2020-04-23 01:54:32.445+0000 [id=18]    INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running

Solution

  • I had exactly same problem on freshly installed Jenkins 2.222.1. It seem that this error is only due to "Validate proxy" button( I think this button does not like when authentication fields are not set) Indeed, when clicking on "submit" button, Jenkins were able to detect they was a newer version of Jenkins and installed it (so I suppose proxy config were ok). After accepting automatic upgrade in Jenkins in version 2.222.3, button "Validate proxy" is working fine now...