From an architecture point of view, when an AWS Step Functions invoke a Lambda, does it go through API Gateway or does the invocation happen directly?
The AWS Step Functions service directly invokes Lambda functions - there is no need for the Step Function service to go via another AWS service to invoke a Lambda.
You can see the lambda:InvokeFunction IAM permission needed by the service, in the AWS Lambda IAM policy template within the Step Function docs. There is no permission required for the API gateway.
You also wouldn't have the option to specify the InvocationType if it went via the API Gateway.