What is a good CI build-process?

Question

What constitutes a good CI build process?

We use CI, but is deployment to production even a realistic CI goal when you have dependencies on several services that should be deployed too and other applications may depend on these too?

Is a good good CI build process good enough when it’s automated to QA and manual from there?

Answer 1

Well "it depends" :)

We use our CI system to:

1. build & unit test
2. deploy to a single box, run integration tests and code analysis
3. deploy to a lab environment
4. run acceptance tests in a production-like system
5. drop builds that pass to code drop for production deployment

This is for a greenfield project of about a dozen services and databases deployed to 20+ servers that also had dependencies on half a dozen other 'external' services.

Using a CI tool to deploy your product to a production environment as a realistic goal? Again, "it depends".

Why would you want to do this?

• if you have the process you can roll changes (and roll back) faster and more often
• less chance for human error
• you can test the same deployment strategy in a test environment before going to production and catch issues earlier

Some technical things you have to address before you can answer this:

• what is the uptime requirements for your system? Are you allowed to have downtime or does it need to be up 24/7?
• do you have change control processes in place that require human intervention/approval?
• is your deployment robust enough for any component to roll back to a known-good state if a deployment fails?
• is your system designed to handle different versions of services or clients in case one or several component deployments fails (and you have the above rollback to last known good)?
• does the process have the smarts to handle a partial deployment where a component cannot handle mixed versions of its dependencies/clients?
• how are you handing database deployment/upgrades?
• do you have monitoring in place so you know when something goes wrong?

Here are a couple of recent related links about automation and building the tools you need.

When it comes down to it the more complex your system the more difficult it is do automate everything, but that does not mean it is not a worthy goal, it just takes a lot more effort and willpower to get it done—everything from knowing the difficulties you're going to face, the problems you have to account for (failure will happen), the political challenges of building infrastructure (vs. more product features).

Now here’s the big secret... the technical challenges are challenging, but not impossible... the political challenges may be insurmountable. Everything about this costs money, whether it’s developer time or buying third-party solutions. So really, can you build the $1K, $10K, $100K, or $1M solution?

Whatever solution you go for, make sure the automation is robust first, complete second... I.e., make sure you have as robust a solution as you can for getting deployment to a test environment rather than a fragile solution that deploys to production.