Search code examples
rubyhttprequestshippable-ci

Ruby unable to call Shippable webhook endpoint

For a while I've been unable to run webhooks from my GitLab instance. At first I thought it was something related to GitLab upgrade ~10.0 release, or some iptables, but now I think it might be more Ruby thing together with how Shippable endpoints are called (in Ruby?).

On the failed request site I can see following information:

  • reason for failure is execution expired
  • URL is https://[username:password]@api.shippable.com/projects/[project id]/newBuild - it's generated by Shippable automatically on enabling project
  • X-Gitlab-Event type is Push Hook
  • there is also JSON with request body

First, I tested wither I can actually connect with Shippable from server

curl --verbose -X POST -H "Content-Type: application/json" -H "X-Gitlab-Event: $event" --data "$json" $url

Request succeeded, which made me think that it is not a matter of iptables (however I checked and no, no iptables rules were set).

Then I attempted to recreate that request inside /opt/gitlab/embedded/bin/irb:

require 'net/http'
require 'net/https'

uri = URI.parse(url)
username = uri.userinfo.split(':')[0]
password = uri.userinfo.split(':')[1]

req = Net::HTTP::Post.new(uri.path, {'Content-Type' =>'application/json', 'X-Gitlab-Event' => event})
req.basic_auth username, password
req.body = json
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true

response = http.start { |http| http.request(req) }

Then it failed just like in GitLab with:

Net::OpenTimeout: execution expired
    from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:880:in `initialize'
    from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:880:in `open'
    from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:880:in `block in connect'
    from /opt/gitlab/embedded/lib/ruby/2.3.0/timeout.rb:101:in `timeout'
    from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:878:in `connect'
    from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
    from /opt/gitlab/embedded/lib/ruby/2.3.0/net/http.rb:852:in `start'
    from (irb):142
    from embedded/bin/irb:11:in `<main>'

Interestingly similar thing happens on my local machine: curl succeeds while Ruby throws.

Additionally I checked out that it shouldn't be a matter of basic auth, SSL nor POST - I successfully POSTed snippet on Bitbucket from my server's irb the same way I tested Shippable webhook endpoint. I even posted the Shippable request on my mock server with virtually the same request format.

At this point I am curious what might be the cause of such behavior and how to debug it further. The only 2 factors I found that were constant in all failing cases is the target (Shippable URI) and the client (Ruby's Net::HTTP). What else do you suggest me to check?

Solution

  • I cannot answer when exactly but the issue disappeared - I assume either GitLab or Shippable update changed something as hooks started working again without me doing any action.