Search code examples
deploymentdevopsaws-cdkcicdaws-codepipeline

How do I properly build multiple pipelines via CDK and CodePipelines in the context of a monorepo?

I've got a monorepo that's structured like this. All infra is CDK v2, and I'm trying to determine the best way to build pipelines in this context.

At this point, I've got two of them fully working, but they're quite slow as they're both triggered when an update is made from either the front or back-end.

my-monorepo/
├── node_modules
├── packages/
│   ├── infra/
│   │   ├── cdk.out
│   │   ├── stacks/
│   │   │   ├── ApiStack.ts
│   │   │   ├── AuthStack.ts
│   │   │   ├── DataStack.ts
│   │   │   ├── LambdaStack.ts
│   │   │   ├── BackendPipelineStack.ts
│   │   │   └── WebAppPipelineStack.ts
│   │   ├── cdk.json
│   │   └── Launcher.ts
│   ├── lib
│   ├── lambda
│   └── web/
│       └── react-vite
├── package.json

Launcher.ts looks like this:

import { App } from "aws-cdk-lib";
import { WebAppPipelineStack } from "./stacks/WebAppPipelineStack";

//CDK App
const app = new App();

//CodePipelines
new BackendPipelineStack(app, "BackendPipelineStack");
new WebAppPipelineStack(app, "WebAppPipelineStack");

app.synth();

The BackendPipelineStack builds out API Gateway, Cognito, a Dynamo table, and a single Lambda that acts as the API.

The WebApPipelineStack simply builds a Vite React app and deploys it.

Both have different inputs, but essentially look the same, creating a self-mutating pipeline:

const pipeline = new CodePipeline(this, "WebAppPipeline", {
    pipelineName: "PrototypeWebAppPipeline",
    publishAssetsInParallel: false,
    synth: new ShellStep("Synth", {
        input: pipelineBuildInput,
        primaryOutputDirectory: "packages/infra/cdk.out",
        commands: [
            "cd packages/infra",
            "npm ci",
            "npx cdk synth"
        ]
    })
});

I get basically what I want - two working pipelines. However, it feels pretty weird - any change pushed from any file in the monorepo triggers both pipelines and in total, takes about 15 minutes to run.

Is this bad practice? What would be a better, simpler design? I can't imagine this scaling well if I were to add more packages to the monorepo and things expand.

Solution

  • CodePipeline executions are triggered by *any* change to the source repo and run *unconditionally* from start to finish. As you experienced, these constraints can result in unnecessary builds, especially in a monorepo. Codepipeline has no native way of selectively deploying only those elements that have changed.

    Is there a "better, simpler design"? That's a matter of opinion, but here are some options:

    1. Dump the monorepo.
    2. Run the backend and frontend deploys in parallel in a single pipeline. Use the Wave construct. This is more complicated if your frontend depends on your backend for inputs.
    3. Build a DIY conditional pipeline. Connect your GitHub repo to a CodeBuild job. Have CodeBuild look at the input artefact and determines which of your monorepo "packages" have changed. If a package has changed, write the artefact to an S3 object (e.g. frontend-build-me.zip). Change your pipelines' source from GitHub to that S3 object. Your pipeline will now only run when frontend-build-me.zip changes in S3. Presto, a conditional pipeline!
    4. Consider Amplify hosting for your frontend CI/CD. It supports features such as branch deploys out of the box. @aws-cdk/aws-amplify-alpha has the constructs you'd need. You can migrate your frontend deploys to Amplify without changing your backend setup.

    Here are some related AWS resources that I've found helpful: