"Insanity: doing the same thing over and over again and expecting different results." - Albert Einstein
I have a vagrant + chef (w/chef-zero Berkshelf) built box that runs a lot of programs, including zend-server-php, mongos, and erlang. I'm using popular cookbooks, managed with Berkshelf, and a basic CentOS 6 box (stored on my local Mac). Firing this box up for the first time (after a vagrant destroy) usually takes multiple attempts. The vagrant up will fail on one of the programs listed above, and I'll have to run vagrant provision over and over (sometimes as many as 6 times) to get through the whole build with no errors. However, on rare occasions the stars align and everything fires up the first try.
The errors I'm seeing are often explicitly timeout errors (though not always). I'm generally running this from a fairly weak office network connection (<20 Mbps down) which might explain why the failures are happening on the largest downloads.
So here's my question: First, is there some other variable at play besides the network connection that could explain why I'd have to vagrant provision multiple times? If so how can I test for and treat it? Second, is there a way to make vagrant keep provisioning until it does so successfully? I haven't found a documented flag/setting for doing so, but it feels like there could be one...
Thanks for your help.
UPDATE:
Here's an example of zend-server-php-5.4 failing:
==> mongos-zend: Running triggers before provision...
the ffi-yajl and yajl-ruby gems have incompatible C libyajl libs and should not be loaded in the same Ruby VM
falling back to ffi which might work (or might not, no promises)
Loading cookbooks from chef-repo/cookbooks
Uploading Cookbook apt
Uploading Cookbook build-essential
Uploading Cookbook cb-platform
Uploading Cookbook chef-sugar
Uploading Cookbook erlang
Uploading Cookbook hostsfile
Uploading Cookbook mongodb
Uploading Cookbook mysql
Uploading Cookbook openssl
Uploading Cookbook percona
Uploading Cookbook postgresql
Uploading Cookbook python
Uploading Cookbook rabbitmq
Uploading Cookbook rbac
Uploading Cookbook resource-control
Uploading Cookbook role-mongodb-configserver
Uploading Cookbook role-mongodb-mongos
Uploading Cookbook role-mongodb-replicaset1
Uploading Cookbook role-mongodb-replicaset2
Uploading Cookbook role-mongodb-shard1
Uploading Cookbook role-mongodb-shard2
Uploading Cookbook role-rabbitmq
Uploading Cookbook role-sphinx
Uploading Cookbook role-twemcache
Uploading Cookbook role-zendserver
Uploading Cookbook runit
Uploading Cookbook smf
Uploading Cookbook sphinx
Uploading Cookbook yum
Uploading Cookbook yum-epel
Uploading Cookbook yum-erlang_solutions
Uploading Cookbook yum-mysql-community
Uploading Cookbook zendserver
==> mongos-zend: Chef 12.0.3 Omnibus package is already installed.
==> mongos-zend: Running provisioner: shell...
mongos-zend: Running: inline script
==> mongos-zend: Running provisioner: chef_client...
==> mongos-zend: Creating folder to hold client key...
==> mongos-zend: Uploading chef client validation key...
Generating chef JSON and uploading...
==> mongos-zend: Running chef-client...
==> mongos-zend: [2015-01-20T10:46:10-06:00] INFO: Forking chef instance to converge...
==> mongos-zend: [2015-01-20T10:46:10-06:00] INFO: *** Chef 11.10.4 ***
==> mongos-zend: [2015-01-20T10:46:10-06:00] INFO: Chef-client pid: 13175
==> mongos-zend: [2015-01-20T10:46:11-06:00] INFO: Setting the run_list to ["recipe[role-mongodb-mongos::default]", "recipe[role-zendserver::default]", "recipe[role-rabbitmq::default]", "recipe[role-twemcache::default]", "recipe[role-sphinx::default]", "recipe[cb-platform::default]"] from JSON
==> mongos-zend: [2015-01-20T10:46:11-06:00] INFO: Run List is [recipe[role-mongodb-mongos::default], recipe[role-zendserver::default], recipe[role-rabbitmq::default], recipe[role-twemcache::default], recipe[role-sphinx::default], recipe[cb-platform::default]]
==> mongos-zend: [2015-01-20T10:46:11-06:00] INFO: Run List expands to [role-mongodb-mongos::default, role-zendserver::default, role-rabbitmq::default, role-twemcache::default, role-sphinx::default, cb-platform::default]
==> mongos-zend: [2015-01-20T10:46:11-06:00] INFO: Starting Chef Run for mongos-zend
==> mongos-zend: [2015-01-20T10:46:11-06:00] INFO: Running start handlers
==> mongos-zend: [2015-01-20T10:46:11-06:00] INFO: Start handlers complete.
==> mongos-zend: [2015-01-20T10:46:11-06:00] INFO: HTTP Request Returned 404 Not Found : Object not found: /reports/nodes/mongos-zend/runs
==> mongos-zend: [2015-01-20T10:46:11-06:00] INFO: Loading cookbooks [apt, build-essential, cb-platform, chef-sugar, erlang, hostsfile, mongodb, mysql, openssl, percona, postgresql, python, rabbitmq, rbac, role-mongodb-mongos, role-rabbitmq, role-sphinx, role-twemcache, role-zendserver, runit, smf, sphinx, yum, yum-epel, yum-erlang_solutions, yum-mysql-community, zendserver]
==> mongos-zend: [2015-01-20T10:46:11-06:00] WARN: CentOS doesn't provide mongodb, forcing use of mongodb-org repo
==> mongos-zend: [2015-01-20T10:46:11-06:00] WARN: 10gen_repo is deprecated, use mongodb_org_repo
==> mongos-zend: [2015-01-20T10:46:11-06:00] WARN: Cloning resource attributes for package[mongo-10gen-server] from prior resource (CHEF-3694)
==> mongos-zend: [2015-01-20T10:46:11-06:00] WARN: Previous package[mongo-10gen-server]: /var/chef/cache/cookbooks/role-mongodb-mongos/recipes/default.rb:15:in `block in from_file'
==> mongos-zend: [2015-01-20T10:46:11-06:00] WARN: Current package[mongo-10gen-server]: /var/chef/cache/cookbooks/mongodb/recipes/install.rb:65:in `from_file'
==> mongos-zend: [2015-01-20T10:46:13-06:00] WARN: Cloning resource attributes for template[/etc/mongodb.conf] from prior resource (CHEF-3694)
==> mongos-zend: [2015-01-20T10:46:13-06:00] WARN: Previous template[/etc/mongodb.conf]: /var/chef/cache/cookbooks/mongodb/recipes/install.rb:14:in `from_file'
==> mongos-zend: [2015-01-20T10:46:13-06:00] WARN: Current template[/etc/mongodb.conf]: /var/chef/cache/cookbooks/mongodb/definitions/mongodb.rb:139:in `block in from_file'
==> mongos-zend: [2015-01-20T10:46:13-06:00] WARN: Cloning resource attributes for package[gcc] from prior resource (CHEF-3694)
==> mongos-zend: [2015-01-20T10:46:13-06:00] WARN: Previous package[gcc]: /var/chef/cache/cookbooks/mongodb/recipes/mongo_gem.rb:3:in `from_file'
==> mongos-zend: [2015-01-20T10:46:13-06:00] WARN: Current package[gcc]: /var/chef/cache/cookbooks/role-zendserver/recipes/default.rb:34:in `block in from_file'
==> mongos-zend: [2015-01-20T10:46:14-06:00] INFO: hostsfile_entry[33.33.33.42] content already matches - nothing to do.
==> mongos-zend: [2015-01-20T10:46:14-06:00] INFO: hostsfile_entry[33.33.33.43] content already matches - nothing to do.
==> mongos-zend: [2015-01-20T10:46:14-06:00] INFO: Starting install for package zend-server-php-5.4
==> mongos-zend: [2015-01-20T10:30:22-06:00] INFO: package[zend-server-php-5.4] installing zend-server-php-5.4-6.2.0-2002 from zend-server repository
==> mongos-zend:
==> mongos-zend: ================================================================================
==> mongos-zend: Error executing action `install` on resource 'package[zend-server-php-5.4]'
==> mongos-zend: ================================================================================
==> mongos-zend:
==> mongos-zend:
==> mongos-zend: Chef::Exceptions::Exec
==> mongos-zend:
==> mongos-zend: ----------------------
==> mongos-zend: returned 1, expected 0
==> mongos-zend:
==> mongos-zend:
==> mongos-zend: Resource Declaration:
==> mongos-zend: ---------------------
==> mongos-zend: # In /var/chef/cache/cookbooks/zendserver/recipes/default.rb
==> mongos-zend:
==> mongos-zend: 65: package package_name do
==> mongos-zend: 66: :install
==> mongos-zend: 67: notifies :restart, 'service[zend-server]', :immediate
==> mongos-zend: 68: end
==> mongos-zend: 69:
==> mongos-zend:
==> mongos-zend:
==> mongos-zend:
==> mongos-zend: Compiled Resource:
==> mongos-zend: ------------------
==> mongos-zend: # Declared in /var/chef/cache/cookbooks/zendserver/recipes/default.rb:65:in `from_file'
==> mongos-zend:
==> mongos-zend: package("zend-server-php-5.4") do
==> mongos-zend: action :install
==> mongos-zend: retries 0
==> mongos-zend: retry_delay 2
==> mongos-zend: package_name "zend-server-php-5.4"
==> mongos-zend: version "6.2.0-2002"
==> mongos-zend: cookbook_name "zendserver"
==> mongos-zend: recipe_name "default"
==> mongos-zend: end
Bandwidth, CPU, I/O, bugs, all could add up to the resource convergence time, which could lead to a timeout. Try increasing the resources to your VM (CPU, RAM), if possible use an SSD on the host , etc.
As for running vagrant provision until it succeeds, you could probably just wrap it in a while loop:
false; while [ "$?" -ne "0" ]; do vagrant provision; done