Search code examples
svnversion-controlreleaseperforcerelease-management

Setting up a complex stage environment with open source tools

I've searched the series of tubes for awhile and haven't found any good answers to this question usually due to a lack of understanding on the part of the question readers of what the use cases are so I am going to be excruciatingly detailed. For example this question: Create a "label" in subversion indicating what files should be in the next release (the 5 vote answer seems close, but not quite there) and this question: Using Subversion Tags to Deploy to Development/Staging/Testing Server are similar to mine except the people attempting to answer don't seem to fully understand the subtleties.

I'm setting up a more sophisticated staging environment for a fast growing project. The current environment consists of a main production branch along with build branches. Builds are not a problem as we can just tag the head revision of the build branch when it's "done" and merge that back into production. The piece that is more subtle is being able to setup an automated process which tags an arbitrary set of files at an arbitrary revision for each file with a label so that you can sync to that label out to staging servers. Now in the SCM world label is overloaded so I will explicitly state that in this case I am using label in the perforce (http://www.perforce.com/perforce/doc.current/manuals/cmdref/label.html) sense of the word meaning a name for a set of files where the files are at arbitrarily chosen revisions and the set is mutable.

So to give a simple example: Suppose I have file A and B. File A's head revision is revision 13 of that file and file B's head revision is revision 4. Currently in production we have A@10 and B@2. The changes to file A have been QAd and has been determined that it's ready to patch out. The first change to file B (revision 3) is ready to patch but the business side has determined that the head change (revision 4) needs to be worked on a little more so it shouldn't patch and gets pushed out to a later date. So for patching to production we need to tag B@3 and A@13 for release. So this is where everyone says "ohh well use tags in ". So that's all well and good for the 20110703 nightly patch. But in the stage environment we also want to be able to test the not-necessarily-head-revision files in the state that would best be described as "if we were to tag the branch right now this is what it would look like" state throughout the day (week, month, etc). Don't get me wrong I don't want to do a bunch of coding in the production branch, but sometimes it's necessary.

The one point I've glossed over so far is that there is also a ticketing system where commits are associated with tasks/tickets and the tasks/tickets are related to a particular release date. So the workflow is that a user creates a task, attaches code to it in the form of changesets (with a one-task-to-many-changesets relationship), the task progresses through an approval process, and eventually is christened as ready to patch. Then there are a series of automated scripts to determine what file revisions are going to patch out on day X and sync the staging environments (or production) to the appropriate versions of files. The specific script that im having trouble with is the one where given a set of tasks, and through them their changesets, that are ready to patch we are able to sync a set of files to the appropriate revisions to emulate what that future production patch will look like. If I were to use perforce I could accomplish this with labels that are mutable and basically just hold a collection of (filename, filerevision) values that a user can sync to. But I'm looking to use open source tools and specifically tools that integrate with Redmine (yes I still need to build the ticket-to-changeset association layer).

So my questions are.

  1. Are there any open-source SCMs that have the concept of a label in them? I've looked a little at mercurial and the queues extension but it, once again, seems to solve a similar but not quite the same problem. (feel free to correct me and say "nope queues solve this perfectly just to this...")
  2. If there aren't any tools that work exactly this way, any suggestions for how to best set this up? I can certainly write a script that kinda fakes labels and manually syncs each individual file but that seems bad in so many ways.

Basically what I'm looking to do is be able to allow actions like a task being moved out a few days or transitioning from a non-patchable to patchable state to be able to affect the state of the code on the staging servers to put them in the "this is what we're planning on patching" state without any human intervention after the task changes.

Thanks for the help.

Solution

  • [This answer attempts to summarise the discussion in the comments above.]

    Modern DVCS (eg. Git, Mercurial) manage changes as sequences of commits, rather than sets of files. Because if this different paradigm, it is difficult to think of a "label" that selects particular files from particular revisions. Commits may touch several files, and commits can both touch a given file (though the change inside the file may or may not overlap).

    To manage a staged release using Git, what you could do is:

    1. Grab the previous release branch.
    2. Pull each branch or cherry-pick each commit that is intended to be in the next staging test (you can do this on a development workstation).
    3. Push to staging repository.
    4. Pull that into the staging server.
    5. When the staging checks out okay, pull that same branch into production.

    In the (hopefully) common case where there aren't any changes you don't want, then step 2 becomes a single-step merge. If you don't want a particular change, then you can cherry-pick the changes you do want.

    Some helpful resources: