Puppet: "Could not find class" hiccups often once after manifest/module changes

I’ve got no idea if this is a bug or some kind of misuse/misconfiguration. Hope somebody can help. Thanks!

If I change a file in a module or manifest directory this often results in errors, mostly of the following kinds

Could not find class for on node
Cannot find definition Class on node

These errors occur when puppet modules/manifests are changed – but it happens only once. If it happens affected puppet agents will use their cached catalog once. The next time these agents request a catalog it is compiled flawlessly and everything will work like a charm – all errors are gone. Problem is we have agents with large catalogs. Their catalog runs can take up to over a minute and so the time it takes for a change to be applied will then be doubled to take up to 2 or even 3 minutes.

Usually changes of modules/manifests are performed by “git pull”. But it’s easy to reproduce these error messages by simple “touch” operations. Here are some interesting log observations and their [commonness]. Yes AFAICS they always appear in triplets. It rarely happens that there are no such problems at all after a modification.

> touch /opt/xxx/dev/puppet/manifests/site.pp

[often] puppet-master[12498]: Could not find class <class> for <node> on <node>
[often] puppet-master[12498]: Could not find class <class> for <node> on <node>
[often] puppet-master[12498]: Could not find class <class> for <node> on <node>

> touch /opt/xxx/dev/puppet/puppet.conf

[often] puppet-master[12498]: Could not find class <class> for <node> on <node>
[often] puppet-master[12498]: Could not find class <class> for <node> on <node>
[often] puppet-master[12498]: Could not find class <class> for <node> on <node>

[sometimes] puppet-master[12498]: Cannot find definition Class on node <node>
[sometimes] puppet-master[12498]: Cannot find definition Class on node <node>
[sometimes] puppet-master[12498]: Cannot find definition Class on node <node>

[rare] puppet-master[12498]: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type <own custom type> at <class file> on node <node>
[rare] puppet-master[12498]: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type <own custom type> at <class file> on node <node>
[rare] puppet-master[12498]: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type <own custom type> at <class file> on node <node>

> touch /opt/xxx/dev/puppet/modules/dispatcher/manifests/init.pp

[often] puppet-master[12498]: Could not find class <class not even part of module dispatcher> for <node not even affected by module dispatcher> on <node not even...>
[often] puppet-master[12498]: Could not find class <class not even part of module dispatcher> for <node not even affected by module dispatcher> on <node not even...>
[often] puppet-master[12498]: Could not find class <class not even part of module dispatcher> for <node not even affected by module dispatcher> on <node not even...>

Affected classes and nodes seem to be randomly chosen.

Environment this runs in:

> ll /etc/puppet
lrwxrwxrwx 1 root root   34 Mar  7 10:01 auth.conf -> /opt/xxx/dev/puppet/auth.conf
lrwxrwxrwx 1 root root   38 Mar  7 10:01 autosign.conf -> /opt/xxx/dev/puppet/autosign.conf
lrwxrwxrwx 1 root root   40 Mar  7 10:01 fileserver.conf -> /opt/xxx/dev/puppet/fileserver.conf
lrwxrwxrwx 1 root root   36 Mar  7 10:01 puppet.conf -> /opt/xxx/dev/puppet/puppet.conf

> ls -A /opt/xxx/dev/puppet/
auth.conf  autosign.conf  fileserver.conf  .git  .gitignore  manifests  modules  .project  puppet.conf  scripts

> cat /etc/puppet/puppet.conf # on puppetmaster
[main]
 logdir = /var/log/puppet
 rundir = /var/run/puppet
 ssldir = $vardir/ssl
 modulepath = /opt/xxx/dev/puppet/modules
 manifestdir = /opt/xxx/dev/puppet/manifests
 manifest = /opt/xxx/dev/puppet/manifests/site.pp

[agent]
 classfile = $vardir/classes.txt
 localconfig = $vardir/localconfig
 server = <puppetmaster host>
 report = true
 splaylimit = 0
 runinterval = 30

[master]
 certname=<puppetmaster host>
 reports = http,log
 reportdir = /var/lib/puppet/reports/upload
 reporturl = http://<puppetmaster host>:3000/reports

> cat /opt/xxx/dev/puppet/manifests/site.pp 
  import 'nodes.pp'
  $puppetserver = <puppetmaster host>

> rpm -qa | egrep "puppet|ruby"
  rubygem-rake-0.8.7-2.1.el6.noarch
  ruby-mysql-2.8.2-1.el6.x86_64
  libselinux-ruby-2.0.94-5.3.el6.x86_64
  puppet-3.1.0-1.el6.noarch
  rubygem-fastthread-1.0.7-2.el6.x86_64
  rubygem-mongrel-1.1.5-3.el6.x86_64
  ruby-1.8.7.352-7.el6_2.x86_64
  ruby-irb-1.8.7.352-7.el6_2.x86_64
  ruby-augeas-0.4.1-1.el6.x86_64
  ruby-shadow-1.4.1-13.el6.x86_64
  puppetlabs-release-6-6.noarch
  rubygems-1.3.7-1.el6.noarch
  puppet-server-3.1.0-1.el6.noarch
  rubygem-gem_plugin-0.2.3-3.el6.noarch
  rubygem-daemons-1.0.10-2.el6.noarch
  puppet-dashboard-1.2.22-1.el6.noarch
  ruby-libs-1.8.7.352-7.el6_2.x86_64
  ruby-rdoc-1.8.7.352-7.el6_2.x86_64
  rubygem-json-1.4.6-1.el6.x86_64

Solution

Switching to an Apache/Passenger/Rack solution (and dumping WEBrick) seems to solve this problem. Additional info: http://projects.puppetlabs.com/issues/19642

Puppet: "Could not find class" hiccups *often* once after manifest/module changes

Puppet: "Could not find class" hiccups often once after manifest/module changes