Chef client re-register breaks ability to save node data back to chef server

Question

When I generate new client keys for a host and put them in /etc/chef/client.pem, my chef-client run goes through fine, except the end state is always marked as "Chef Client failed." I believe this is the node data that the node tries to update for itself on the chef-server, and somehow that data requires the old client.pem file.

To give more context:

1. My host disk dies and it gets re-imaged. Once the OS is installed, I need to bootstrap it with chef.
2. I would rather leave the roles associated with this node untouched. As such, my automation that kicks in after host installation copies over (from a secure server) the validation.pem, a headless_admin.pem and does a rest 'DELETE /clients/clientname'. I do not 'DELETE /nodes/nodename' as I want to preserve roles associated with the nodename.
3. Running chef-client now generates a new client.pem file (similar to doing a 'knife client reregister clientname' from a workstation.
4. However, though this chef-client run converges as expected, it bails out at the end with a failed state. Enabling debug mode doesn't provide any more info. I strongly suspect this is failing as node tries to update its saved state on the chef server.

Is there a way to enable a client to ignore any changes to the client.pem and update its node state. (without that, the whole concept of re-registering seems pointless).

Thanks for reading through and any insights on this

Answer 1

Assuming you are using either Enterprise Chef or the new Chef Server 12, you need to fix the ACLs on the node object. By default when the client first creates its own node object, it inherits write access to it. In the old Open Source Server (11) this check is simpler, where the names of the client and node just have to match, but the ACL system is a bit more complex. You can likely fix this via knife-acl.