AWS EventBridge Pipes doesn't start execution of express state machine

Question

I want to start an AWS Step Functions express state machine whenever messages are available in AWS SQS.

Using the aws-cdk, I configured the necessary resources

const stateMachine = new StateMachine(this, 'StateMachine', {
    definitionBody: DefinitionBody.fromChainable(someTask),
    stateMachineType: StateMachineType.EXPRESS,
    logs: {
        destination: new LogGroup(this, 'LogGroup'),
        includeExecutionData: true,
        level: LogLevel.ALL,
    },
})

const pipeRole = new Role(this, 'PipeRole', {
    assumedBy: new ServicePrincipal('pipes.amazonaws.com'),
})

queue.grantConsumeMessages(pipeRole)
stateMachine.grantStartExecution(pipeRole)

const pipe = new CfnPipe(this, 'Pipe', {
    roleArn: pipeRole.roleArn,
    source: queue.queueArn,
    target: stateMachine.stateMachineArn
})

but the state machine is never executed when I send new messages to the queue.

There are failed invocation events in CloudWatch but no error messages (I also tried using AWS CloudTrail).

To see if the Pipe ran at all, I added an enrichment Lambda and observed the logs.

When I create the EventBridge Pipe through the web console it works.

Answer 1

The problem here is that AWS EventBridge Pipe, by default, uses the synchronous invocation type for the express state machines. However, stateMachine.grantStartExecution(pipeRole) only allows a StartExecution action but not a StartSyncExecution, hence, EventBridge Pipe is missing the permissions to invoke the state machine.

There are at least two possible solutions:

1. Change the invocation type to asynchronous, i.e.

const pipe = new CfnPipe(this, 'Pipe', {
    roleArn: pipeRole.roleArn,
    source: props.extractCompanyWebsiteQueue.queueArn,
    target: stateMachine.stateMachineArn,
    targetParameters: {
        stepFunctionStateMachineParameters: {
            invocationType: 'FIRE_AND_FORGET',
        },
    },
})

or

2. Replace stateMachine.grantStartExecution(pipeRole) with stateMachine.grantStartSyncExecution(pipeRole)

If you are not using aws-cdk, you can fix the inline policy of the EventBridge Pipe's role by adding:

 {
    "Action": "states:StartExecution",
    "Resource": "<arn of state machine>",
    "Effect": "Allow"
}

to the Statements array.

Note: you also need to lookout when using input templates (transformers) in AWS EventBridge Pipe: It may be that your output is not valid JSON, and a state machine won't run but won't give an error message!