Hello I've installed Gitalb CE (v9.3.8) on my private server (Debian Stretch) and I have got myself into a problem with Sidekiq. Aparently it is trying to load an invalid JSON, but I don't know from where. Do you have any idea, where could have the "nalse" (second line in the log below) come from?
2017-09-06_12:23:27.91969 2017-09-06T12:23:27.919Z 27771 TID-ot0tpp4p4 INFO: Booting Sidekiq 5.0.0 with redis options {:path=>"/var/opt/gitlab/redis/redis.socket", :namespace=>"resque:gitlab", :url=>nil}
2017-09-06_12:23:27.95895 784: unexpected token at '{"retry":nalse,"queue":"cronjob","class":"StuckCiBuildsWorker","args":[]}'
2017-09-06_12:23:27.95908 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/json-1.8.6/lib/json/common.rb:155:in `parse'
2017-09-06_12:23:27.95911 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/json-1.8.6/lib/json/common.rb:155:in `parse'
2017-09-06_12:23:27.95913 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq.rb:171:in `load_json'
2017-09-06_12:23:27.95916 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-cron-0.6.0/lib/sidekiq/cron/job.rb:289:in `initialize'
2017-09-06_12:23:27.95918 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-cron-0.6.0/lib/sidekiq/cron/job.rb:213:in `new'
2017-09-06_12:23:27.95921 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-cron-0.6.0/lib/sidekiq/cron/job.rb:213:in `block in all'
2017-09-06_12:23:27.95924 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-cron-0.6.0/lib/sidekiq/cron/job.rb:211:in `collect'
2017-09-06_12:23:27.95926 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-cron-0.6.0/lib/sidekiq/cron/job.rb:211:in `all'
2017-09-06_12:23:27.95929 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-cron-0.6.0/lib/sidekiq/cron/job.rb:488:in `destroy_removed_jobs'
2017-09-06_12:23:27.95931 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-cron-0.6.0/lib/sidekiq/cron/job.rb:162:in `load_from_hash!'
2017-09-06_12:23:27.95936 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/sidekiq.rb:41:in `block in <top (required)>'
2017-09-06_12:23:27.95939 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq.rb:73:in `configure_server'
2017-09-06_12:23:27.95941 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/sidekiq.rb:8:in `<top (required)>'
2017-09-06_12:23:27.95943 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:268:in `load'
2017-09-06_12:23:27.95946 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:268:in `block in load'
2017-09-06_12:23:27.95948 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:240:in `load_dependency'
2017-09-06_12:23:27.95951 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:268:in `load'
2017-09-06_12:23:27.95954 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/engine.rb:652:in `block in load_config_initializer'
2017-09-06_12:23:27.95956 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications.rb:166:in `instrument'
2017-09-06_12:23:27.95961 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/engine.rb:651:in `load_config_initializer'
2017-09-06_12:23:27.95964 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
2017-09-06_12:23:27.95971 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/engine.rb:615:in `each'
2017-09-06_12:23:27.95974 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/engine.rb:615:in `block in <class:Engine>'
2017-09-06_12:23:27.95976 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:30:in `instance_exec'
2017-09-06_12:23:27.95979 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:30:in `run'
2017-09-06_12:23:27.95981 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:55:in `block in run_initializers'
2017-09-06_12:23:27.95983 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:228:in `block in tsort_each'
2017-09-06_12:23:27.95985 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
2017-09-06_12:23:27.95988 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
2017-09-06_12:23:27.95993 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:431:in `each_strongly_connected_component_from'
2017-09-06_12:23:27.95995 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
2017-09-06_12:23:27.95997 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:44:in `each'
2017-09-06_12:23:27.96000 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:44:in `tsort_each_child'
2017-09-06_12:23:27.96002 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:415:in `call'
2017-09-06_12:23:27.96004 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:415:in `each_strongly_connected_component_from'
2017-09-06_12:23:27.96006 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:349:in `block in each_strongly_connected_component'
2017-09-06_12:23:27.96009 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:347:in `each'
2017-09-06_12:23:27.96011 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:347:in `call'
2017-09-06_12:23:27.96013 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:347:in `each_strongly_connected_component'
2017-09-06_12:23:27.96016 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:226:in `tsort_each'
2017-09-06_12:23:27.96018 /opt/gitlab/embedded/lib/ruby/2.3.0/tsort.rb:205:in `tsort_each'
2017-09-06_12:23:27.96022 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:54:in `run_initializers'
2017-09-06_12:23:27.96025 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.8/lib/rails/application.rb:352:in `initialize!'
2017-09-06_12:23:27.96027 /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>'
2017-09-06_12:23:27.96029 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `require'
2017-09-06_12:23:27.96032 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `block in require'
2017-09-06_12:23:27.96034 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:240:in `load_dependency'
2017-09-06_12:23:27.96037 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `require'
2017-09-06_12:23:27.96039 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/cli.rb:252:in `boot_system'
2017-09-06_12:23:27.96041 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/cli.rb:54:in `run'
2017-09-06_12:23:27.96046 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/bin/sidekiq:12:in `<top (required)>'
2017-09-06_12:23:27.96048 /opt/gitlab/embedded/bin/sidekiq:22:in `load'
2017-09-06_12:23:27.96050 /opt/gitlab/embedded/bin/sidekiq:22:in `<top (required)>'
2017-09-06_12:23:27.96052 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `load'
2017-09-06_12:23:27.96055 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `kernel_load'
2017-09-06_12:23:27.96061 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in `run'
2017-09-06_12:23:27.96063 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in `exec'
2017-09-06_12:23:27.96065 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
2017-09-06_12:23:27.96068 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
2017-09-06_12:23:27.96070 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
2017-09-06_12:23:27.96072 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in `dispatch'
2017-09-06_12:23:27.96078 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
2017-09-06_12:23:27.96080 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in `start'
2017-09-06_12:23:27.96082 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:34:in `block in <top (required)>'
2017-09-06_12:23:27.96085 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in `with_friendly_errors'
2017-09-06_12:23:27.96087 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:26:in `<top (required)>'
2017-09-06_12:23:27.96089 /opt/gitlab/embedded/bin/bundle:22:in `load'
2017-09-06_12:23:27.96091 /opt/gitlab/embedded/bin/bundle:22:in `<main>'
Faulty client? Sidekiq just basically reads a JSON of a que, something put a faulty JSON inside, and it probably wasn't sidekiq, but something that interacts with it.
Check out if something in your code inserts the Sidekiq jobs "manually" - there were instructions once upon a time how to push sidekiq jobs from other frameworks: https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/client.rb#L69