I am having a problem with codeclimate-test-reporter and Travis CI on my new project. The coverage analysis on Travis CI is completely different from what it is locally. Specifically, the LOC locally is 56 (which is accurate), but it's listed as 10,333 when I build the project on Travis CI.
Local Output (56 / 56 LOC)
Finished in 0.0749 seconds (files took 0.60563 seconds to load)
7 examples, 0 failures
Coverage report generated for RSpec to /Users/lee/GitHub/rescuetime/coverage.
56 / 56 LOC (100.0%) covered.
Coverage = 100.0%. Sending report to https://codeclimate.com for branch
data-analytics-api... done.
Travis CI Output (4954 / 10333 LOC?!)
Coverage report generated for RSpec to /home/travis/build/leesharma/rescuetime/coverage.
4954 / 10333 LOC (47.94%) covered.
Coverage = 47.94%. Sending report to https://codeclimate.com for branch
data-analytics-api... done.
spec_helper.rb (bits relating to codeclimate-test-reporter)
require 'codeclimate-test-reporter'
require 'simplecov'
SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter[
SimpleCov::Formatter::HTMLFormatter,
CodeClimate::TestReporter::Formatter
]
SimpleCov.start
...
VCR.configure do |config|
...
config.ignore_hosts 'codeclimate.com'
end
I've confirmed that the code is exactly the same, and these results are the same for each ruby tested (both locally and on Travis CI, >=1.9.3 MRI). I tried limiting it to just one ruby version for Travis, but the results are the same. I've tried searching for similar issues, but so far, I've only found an open/unanswered GitHub issue on their repo.
Honestly, I'm not even sure what to try next for debugging this since I can't reproduce it locally. Any idea what to try next?
I sent an email to CodeClimate, and they replied with an extremely helpful response!
I was calling SimpleCov.start instead of CodeClimate::TestReporter.start, which meant that my /vendor and /spec directories were being included in the analysis.
I needed to replace my test coverage reporters section of spec_helper.rb with:
# Test coverage reporters
require 'codeclimate-test-reporter'
CodeClimate::TestReporter.start
SimpleCov.formatter = SimpleCov::Formatter::MultiFormatter[
SimpleCov::Formatter::HTMLFormatter,
CodeClimate::TestReporter::Formatter
]
Hey Lee-
Thanks for the detailed email. I did some digging on my end, and I believe I see what's happening here. This comes down to different exclusions being applied in different testing environments. Broadly speaking, the reason you're seeing such a high LOC in Travis is because coverage data for files in your /vendor and /spec folder ARE being included in the LOC count. I've explained below how to exclude these directories when Travis runs your builds and POSTs Code Climate coverage data.
To investigate this, I first looked over your spec_helper.rb file in GitHub, and I noticed one thing that was different from what Code Climate typically requires. Specifically, I saw that SimpleCov.start was being called instead of CodeClimate::TestReporter.start. That can cause a number of different problems, one of which is that the file exclusions we typically apply aren't triggered. That is, normally our reporter auto-excludes coverage data from all /vendor and /spec directories -- however, if you don't call CodeClimate::TestReporter.start then our exclusions won't kick in. In most cases, you won't want these directories included, since they don't represent code that you've written.
To correct this, I went ahead and forked your repository, and then added my fork to Travis. I then made some changes to the project's spec_helper.rb file, as seen here: https://github.com/jonathancadepowers/rescuetime/blob/master/spec/spec_helper.rb Note that I'm starting the Code Climate reporter (on Line 3) and that I'm no longer starting SimpleCov directly. Since Code Climate's Ruby test reporter is actually just a thin wrapper around SimpleCov, you don't need to directly start SimpleCov (the HTML report from SimpleCov should still get generated each time you run your tests).
After making my changes, I ran a build in Travis. Since your .travis.yml lists four different versions of Ruby, this resulted in four separate build jobs. For all of them, the line count dropped down to 30 (since our exclusions are now triggered) and you now have 100% coverage. See Line 152 here: https://travis-ci.org/jonathancadepowers/rescuetime/jobs/59266389#L152
My Travis build POSTed your Code Climate instance coverage data, so you now have 100% coverage there as well: https://codeclimate.com/github/leesharma/rescuetime
Of course, 56 doesn't equal 30. That said, I think I see why that's the case as well. I noticed from earlier payloads that Travis had sent us for your project that not only were /vendor files being included in the coverage payloads, but coverage data for files in your /spec folder were included as well. Digging in deeper, I saw that two specific /spec files were included: spec/unit/activities_spec.rb (11 total lines) and spec/unit/client_spec.rb (15 total lines). So: 56 - 11 - 15 = 30. That is, I would bet that locally /vendor is being excluded when you run your tests, but /spec is not. If you make the spec_helper.rb changes that I recommended above, both /vendor and /spec will be filtered out, taking you down to 30. So, 30 is probably the correct number here, not 56.
You should just be able to make the spec_helper.rb changes that I mentioned above to correct this issue. Of course, if there's anything else that I can help with, just let me know. Sorry again for the confusion here.
Thanks,
-Jonathan