Error from server (BadRequest): container "microsercvice-registry" in pod "microse...." is waiting to start: trying and failing to pull image

Question

I have a helm chart which is using aws ecr to pull images and deploy them into native Kubernetes:

Chart.yml:

apiVersion: v2
description: helm chart for microservice-registry
name: microservice-registry
type: application
version: 0.0.1

values.yaml

namespace:

replica: 1

labels:
 app: microservice-registry

imagePullSecrets: microservice-registry-secret

image:
 repository: 123456789.dkr.ecr.us-east-1.amazonaws.com/microservice-registry
 tag: 0.0.1
 imagePullPolicy: Always
 
strategy:
 rollingUpdate:
  maxSurge: 1
  maxUnavailable: 0
 type: RollingUpdate

service:
 containerPort: 8761
 port: 80
 type: NodePort
 nodePort: 30061

secrets:
 data: 
   .dockerconfigjson: 123456789
 type: kubernetes.io/dockerconfigjson

awscreds:
 data:
  SECRET_KEY: 123456789
  ACCESS_KEY: 123456789
 name: awscreds-registry
 region: us-east-1

cronjob:
 name: microservice-registry-job
 image: gtsopour/awscli-kubectl:latest

When I deploy the helm chart I get:

root@node1:~# kubectl get pods
NAME                                               READY   STATUS             RESTARTS        AGE
microservice-registry-7c4654467c-q292p             0/1     ImagePullBackOff   0               17m
cicd                                               1/1     Running            1 (22d ago)     44d
grafana-f6d989bbf-hrr6c                            1/1     Running            1 (22d ago)     45d
ingress-nginx-controller-6b8bfd7f69-cwd2b          1/1     Running            1 (22d ago)     35d
nfs-subdir-external-provisioner-5f7dff96d9-rj75j   1/1     Running            39 (2d3h ago)   38d
zookeeper-0                                        1/1     Running            1 (22d ago)     38d
root@node1:~#

log:

root@node1:~# kubectl logs microservice-registry-7c4654467c-q292p
Error from server (BadRequest): container "microservice-registry" in pod "microservice-registry-7c4654467c-q292p" is waiting to start: trying and failing to pull image
root@node1:~#

I have a similar service which is using the same aws ecr credentials. It's deployed successfully. Do you know how I can fix this issue?

Policy JSON:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetAuthorizationToken"
            ],
            "Resource": "*"
        }
    ]
}

Answer 1

According to your https://pastebin.com/7x3NjD8s, you got 401 Unauthorized from ECR. Since you have an On-Premises Kubernetes instance you need to use a process to set your AWS credentials on ImagePullSecrets on your PODs. I would use this one: https://github.com/upmc-enterprises/registry-creds, it will collect your AWS credentials and update the ImagePullSecrets automatically with token to login on AWS ECR.