puppet apply. error: cannot generate tempfile `/tmp/puppet...'

One one of the servers with Centos I am getting error with puppet apply:

puppet apply \
--verbose \
--modulepath=modules \
--node_terminus=exec \
--external_nodes=$PWD/modules/puppet/files/hostinfo/nodeclassifier.rb \
--color=true \
--no-daemonize \
--no-usecacheonfailure \
--server=notahost \
--config=$PWD/modules/puppet/files/puppet.conf \
--debug \
manifests/site.pp
debug: Executing '/opt/puppet-data/puppet/modules/puppet/files/hostinfo/nodeclassifier.rb [host name]'
Could not run: cannot generate tempfile `/tmp/puppet20230712-12431-18iovt1-9'

Also:

ls -la /
total 3105
drwxr-xr-x.    26 root root    4096 Jan 12 09:22 .
drwxr-xr-x.    26 root root    4096 Jan 12 09:22 ..
...
drwxrwxrwt. 32000 root root 3035136 Jul 12 08:23 tmp
drwxr-xr-x.    15 root root    4096 Sep 14  2014 usr
drwxr-xr-x.    24 root root    4096 Mar 18  2021 var

Filesystem is RW:

[root@host puppet]# echo "Test" > /tmp/puppet20230712-4607-ywx98g-9
[root@host puppet]#

And folder attributes:

# lsattr /
...
----------I---- /tmp
...

Solution

Your /tmp directory has 32000 subdirectories. This is the limit for ext3, and maybe for other filesystems, too. If the file Puppet is trying to create is in fact a directory, then it is not surprising that the attempt fails. Other software will be affected, too.

Cleaning out your /tmp folder should resolve the issue.

You should also try to determine why your /tmp filled so full in the first place, and take appropriate measures to prevent that from happening again. You could consider Puppet's Tidy resource type for that, or the systemd-tmpfiles service, or perhaps modifications to whatever software is generating subdirectories in /tmp and failing to clean them up.