Is there a way through which I can check the CPU Utilization of each of the containers for all the tasks in a service? I have containerInsights enabled on the cluster.
If you have containerInsights enabled on the cluster, then those metrics you are asking about should be in CloudWatch. You will also see them in the Container Insights dashboard. I suggest reading the Container Insights release announcement that shows examples of the metrics and the dashboard.