I want to use AWS SQS for communication between my microservices (and later possibly SNS). Each microservice can have multiple instances up.
Currently I'm trying to implement the Request/Response pattern of message queues.
As I understand it, the normal way is to have one request queue, and pass a unique response queue per service instance.
The consuming service will process the message and send the response to the given response queue. Thus, the response will always be returned to the correct instance of the requesting service.
My problem now comes with Cloudfoundry.
How it should work:
Service A needs to request data from Service B.
There is one queue named A-request-B.
Service A starts with 6 instances.
Every instance creates its own queue: B-response-A-instance[x]
Every request from an instance of A sends their response queue name in the request so response is routed to the correct queue.
This is the only way I know to guarantee that the response from B gets to the correct instance of A.
This doesn't work as Cloudfoundry doesn't allow the "create-queue" call from SQS, even if I can connect to the SQS instance to send and receive messages.
The only way to create a queue is via the command line.
So I would have to create these 6 response-queues manually beforehand.
And if I start a 7th instance of A, it will fail as it doesn't have its own response queue.
I also tried using the SQS temporary queues, but they also work by creating queues dynamically which is not possible in Cloudfoundry.
I'm currently stuck with SQS, so switching to kafka/rabbitmq or something else is not possible.
Is there any other way to pass a response to the matching service instance? Or is there another way to create queues in cloud foundry?
Summary from comments above...
This doesn't work as Cloudfoundry doesn't allow the "create-queue" call from SQS
Cloud Foundry doesn't really care what messaging system you're using, unless you're using a Marketplace service to create it. In that case, Cloud Foundry will work on your behalf to create a service instance. It does this by talking to a service broker, which does the actual creation of the service instance and user credentials.
In your case, Cloud foundry handles creating the credentials to the AWS SQS through the AWS Service Broker. Unfortunately, the credentials the broker gives you don't have the permission to create queues. The creds are only allowed to send and receive messages for the specific queue that was created by the broker.
There's not a lot you can do about this, but there's a couple options:
Don't use the Marketplace service. Instead, just go to AWS directly, create an IAM user, create your SQS resources, and give the IAM user permissions to them.
Then create a user provided service with the credentials and information for the resources you created. You can bind the user provided service to your apps just like a service created by the AWS Service broker. You'd lose the convenience of using the broker, but you won't have to jump through the hoops you listed when scaling up/down your app instances.
You could create a service instance through the broker, then create a service key. The service key is a long-lived set of credentials so you could then go into AWS, look up the IAM user associated with that service key and adjust the permissions so that you can create queues.
You would then need to create a user provided service, like the first option, insert the credentials and information for your service key and bind the user provided service to any apps that you'd like to use that service.
Don't delete the service key, or your modified user will be removed and your user provided service will stop working.
Hope that helps!