I'm trying to migrate some batch processes from a corporate network to AWS Fargate / ECS.
The process I'm working on now is a containerised Python program which takes some command line arguments. At the moment it's triggered by Jenkins. There's a job definition that calls the same container about 50 times, each with slightly different arguments. When all of these jobs have executed the batch is "done".
At the moment all of these jobs are executing on company internal servers which are a bit slow. I'd like to have all the execution take place on AWS Fargate instead.
I've built an image for my project which is hosted on ECR and I've been able to run that image by creating a task definition and then adding that task to a service in a cluster.
It all seems to work well so far and the performance is much better than running on company iron.
The next thing I'd like to be able to do is run that task from the AWS cli-tool, but this time specify some command line arguments. Can that be done? Is there a way to say - "please run this task definition, but use these command-line arguments instead?"
I've noticed there's an aws command called "ecs start-task", but that requires a container ID. I'm not sure where to find this information. Is start task the thing I need to run a previously defined task definition?
Also, since I'd like to use this in place of an existing process, is there a way to make my AWS CLI command block until the remotely executing task has completed?
Edit 1:
I'm trying the "run-task" command, but the documentation isn't very clear on what value I need to provide for network configuration? This is just a batch so doesn't need to accept any inbound connections at all. It will connect to some other databases:
>aws ecs run-task --task-definition fv-scf-0 --cluster fv-sal
An error occurred (InvalidParameterException) when calling the RunTask operation: Network Configuration must be provided when networkMode 'awsvpc' is specified.
Edit 2:
I think I've got figured out what's needed for the Networking config, but now how do I get past this next error?
>aws ecs run-task --task-definition fv-scf-0 --cluster fv-sal --network-configuration "awsvpcConfiguration={subnets=[subnet-x,subnet-y
5],securityGroups=[sg-z]}"
An error occurred (InvalidParameterException) when calling the RunTask operation: No Container Instances were found in your cluster.
You would use the aws ecs run-task command to run a new container from the command line. Inside the overrides parameter you would pass a command value, which overrides the command that is run when the container starts.
ECS Container Overrides are documented here.