Infrastructure as Code: How do you reconcile or balance mutable vs immutable aspects

Question

We're on our migration path to IaC and I buy into the immutable concept. But my team is asking good questions, like: "you want me to redeploy an entire fleet of VMs just because we need to make a small change? Isn't that overkill? It would be faster if I just ran some remote Powershell to implement a change."

And I get the benefits of immutability. But the questions are pushing me to wonder if we should refine our concept of immutable. In our case, we're looking at Saltstack with Packer building immutable images. A few dozen VMs (plus other infrastructure items that are out of scope for this question). Salt makes having immutable and mutable IaC easy (the latter via applying states). But whatever your tool, where do you draw the line?

• Do we go 100% immutable? Every change requires throwing out an old VM and ushering in a new one. No applying changes via states (or any manual method)
• Or do we have a hybrid? Possibly a very basic "base" immutable image and everything else gets applied mutably? (Via saltstate in my case.) Losing some benefits of immutability but gaining runtime agility? Note that we'd still have change management in place, so it's not like we lose track of our states.

The answer here may be "it depends." But I'd love to hear if anyone has good strategies that make sense and can be applied consistently. What belongs in the immutable Packer image? What is owned by salt states?

(I realize this is basically an orchestration vs config management argument, but I'm already invested in the former; my question is how much, if any, of the latter to pepper in and what are successful strategies for that. I've read seemingly every article out there on combining the two but none with any good scripture.)

Answer 1

Going to give a half answer to my own question, to close this out.

If you read the internet, immutability is the holy grail of infrastructure and there are only good things to come from it. Everyone on Earth and Mars wants it. But the truth is that not everyone really needs it, or it doesn't work for them (tbf, there are a lot of articles out there saying that you need to weigh the benefits/costs with that strategy). Below are the points we used in our solution:

• Immutability is a goal. It's a goal we will never get to 100% on, and if you're starting from scratch in the IaC space, you probably shouldn't set your goals too high on this. We will move more things to an immutable space over time. It's ok to start with 1% immutability.
• There is nothing wrong with change management and pushing IaC changes out via something like Salt states. In many cases, this is already going to be a huge improvement over what people in my boat have.
• We started with the simplest "base" images created by Packer, installing other apps as needed via Salt. Over time, more of those apps will be built into those Packer images. The key is first building a manageable platform for IaC, and the easiest way to do that is via config management, if you have any sort of existing infrastructure to start with.

So in the end, a hybrid approach seems to work best for us, and I think it will for most shops out there. Unless you have a massive team dedicated to maintaining your IaC setup, a completely immutable platform is likely unrealistic. I hope this helps anyone that was in the same place I was.