Kubernetes ErrImagePull - Using Docker Private Registry

서론

이번 포스팅에서는 Kubernetes에 Deployment Pod를 생성할 때 Docker Image를 Pull 해 오지 못하는 현상에 대해 알아보겠습니다.

일반적으로 해당 현상은 Docker Private Registry에 Push 된 이미지의 이름이나, Tag가 없을 경우에 발생할 수 있지만, 본 포스팅에서는 이와는 다른 한가지 현상에 대해 다루려고 합니다.

1. 문제 발생

배포하고자 하는 이미지는 httpd/2.4.41 dockerhub 이미지를 기준으로 간단히 커스터마이징한 이미지를 사용합니다.

해당 이미지를 내부에 구축해 둔 저장소에 이미지를 push하고 아래와 같이 deployment.yaml 파일을 작성합니다.

apiVersion: apps/v1 kind: Deployment metadata:   name: httpd-deployment   labels:     app: httpd spec:   replicas: 1   selector:     matchLabels:       app: httpd   template:     metadata:       labels:         app: httpd     spec:       containers:       - name: httpd         image: 192.168.56.100:5000/middleware/httpd/httpd_custom_image:2.4.41         imagePullPolicy: Always         ports:         - containerPort: 80 --- apiVersion: v1 kind: Service metadata:   name: httpd spec:   selector:     app: httpd   ports:     - protocol: TCP       port: 80       targetPort: 80   type: ClusterIP --- apiVersion: extensions/v1beta1 kind: Ingress metadata:   name: httpd-ingress   namespace: default spec:   rules:   - host: test.nrson.co.kr     http:       paths:       - backend:           serviceName: httpd           servicePort: 80         path: /

해당 yaml 파일에는 deployment, service, ingress를 정의하였습니다.

[root@nrson kubernetes_deploy]# kubectl create -f deployment.yaml  deployment.apps/httpd-deployment created service/httpd created ingress.extensions/httpd-ingress created [root@nrson kubernetes_deploy]#

위와 같이 deployment, service, ingress를 각각 생성하고 생성된 정보를 확인해 보겠습니다.

[root@nrson kubernetes_deploy]# kubectl get pods --all-namespaces -o wide NAMESPACE     NAME                                      READY   STATUS             RESTARTS   AGE     IP               NODE         NOMINATED  NODE   READINESS GATES default       httpd-deployment-5fd659bb4d-wsjfm         0/1     ImagePullBackOff   0          3m53s   10.233.104.12    kubeworker               kube-system   calico-kube-controllers-cbbd89c57-6ffnw   1/1     Running            1          9h      192.168.56.102   kubeworker               kube-system   calico-node-cpcgj                         1/1     Running            3          9h      192.168.56.102   kubeworker               kube-system   calico-node-ltwdb                         1/1     Running            5          9h      192.168.56.101   kubemaster               kube-system   coredns-58687784f9-md5rs                  1/1     Running            1          9h      10.233.116.3     kubemaster               kube-system   coredns-58687784f9-mjrkg                  1/1     Running            1          9h      10.233.104.5     kubeworker               kube-system   dns-autoscaler-79599df498-4vgk2           1/1     Running            1          9h      10.233.116.4     kubemaster               kube-system   kube-apiserver-kubemaster                 1/1     Running            3          9h      192.168.56.101   kubemaster               kube-system   kube-controller-manager-kubemaster        1/1     Running            1          9h      192.168.56.101   kubemaster               kube-system   kube-proxy-5tkkz                          1/1     Running            1          9h      192.168.56.102   kubeworker               kube-system   kube-proxy-jlxhd                          1/1     Running            3          9h      192.168.56.101   kubemaster               kube-system   kube-scheduler-kubemaster                 1/1     Running            1          9h      192.168.56.101   kubemaster               kube-system   kubernetes-dashboard-556b9ff8f8-w8cfq     1/1     Running            2          9h      10.233.104.4     kubeworker               kube-system   nginx-proxy-kubeworker                    1/1     Running            1          9h      192.168.56.102   kubeworker               kube-system   nodelocaldns-4wmkz                        1/1     Running            1          9h      192.168.56.101   kubemaster               kube-system   nodelocaldns-vvkjj                        1/1     Running            1          9h      192.168.56.102   kubeworker               [root@nrson kubernetes_deploy]# kubectl get service --all-namespaces -o wide NAMESPACE     NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE     SELECTOR default       httpd                  ClusterIP   10.233.19.128           80/TCP                   4m17s   app=httpd default       kubernetes             ClusterIP   10.233.0.1              443/TCP                  9h       kube-system   coredns                ClusterIP   10.233.0.3              53/UDP,53/TCP,9153/TCP   9h      k8s-app=kube-dns kube-system   kubernetes-dashboard   ClusterIP   10.233.60.80            443/TCP                  9h      k8s-app=kubernetes-dashboard [root@nrson kubernetes_deploy]# kubectl get ingress --all-namespaces -o wide NAMESPACE   NAME            HOSTS              ADDRESS   PORTS   AGE default     httpd-ingress   test.nrson.co.kr             80      4m23s [root@nrson kubernetes_deploy]#

위와 같이 httpd-deployment pods로 생성된 [httpd-deployment-5fd659bb4d-wsjfm]의 STATUS가 ImagePullBackOff 상태임을 알 수 있습니다.

확인을 위해 kubectl logs 명령어를 통해 pods 로그를 확인해 보겠습니다.

[root@nrson kubernetes_deploy]# kubectl logs -f httpd-deployment-5fd659bb4d-wsjfm Error from server (BadRequest): container "httpd" in pod "httpd-deployment-5fd659bb4d-wsjfm" is waiting to start: trying and failing to pull image [root@nrson kubernetes_deploy]#

로그 상에서도 이미지를 Pull 해오지 못하는 현상처럼 보입니다.

2. 증상 분석

이를 확인하기 위해 docker에서 docker pull로 직접 이미지를 당겨보겠습니다.

[root@nrson kubernetes_deploy]# docker pull 192.168.56.100:5000/middleware/httpd/httpd_custom_image:2.4.41 2.4.41: Pulling from middleware/httpd/httpd_custom_image 000eee12ec04: Pull complete  32b8712d1f38: Pull complete  f1ca037d6393: Pull complete  c4bd3401259f: Pull complete  51c60bde4d46: Pull complete  3204d6a2792a: Pull complete  Digest: sha256:336eb3ff895207a7af5d2eceff2699979fb076d416334ff1ced29d3dddd15274 Status: Downloaded newer image for 192.168.56.100:5000/middleware/httpd/httpd_custom_image:2.4.41 192.168.56.100:5000/middleware/httpd/httpd_custom_image:2.4.41 [root@nrson kubernetes_deploy]#

역시나 이미지는 잘 당겨 오는 것을 볼 수 있습니다.

이슈 상황을 보다 상세히 살펴보기 위해 다음 정보를 확인합니다.

[root@nrson kubernetes_deploy]# kubectl describe pods httpd-deployment-5fd659bb4d-wsjfm Name:         httpd-deployment-5fd659bb4d-wsjfm Namespace:    default Priority:     0 Node:         kubeworker/192.168.56.102 Start Time:   Sun, 15 Dec 2019 01:28:12 +0900 Labels:       app=httpd               pod-template-hash=5fd659bb4d Annotations:   Status:       Pending IP:           10.233.104.12 IPs:   IP:           10.233.104.12 Controlled By:  ReplicaSet/httpd-deployment-5fd659bb4d Containers:   httpd:     Container ID:        Image:          192.168.56.100:5000/middleware/httpd/httpd_custom_image:2.4.41     Image ID:            Port:           80/TCP     Host Port:      0/TCP     State:          Waiting       Reason:       ImagePullBackOff     Ready:          False     Restart Count:  0     Environment:         Mounts:       /var/run/secrets/kubernetes.io/serviceaccount from default-token-s7jhb (ro) Conditions:   Type              Status   Initialized       True    Ready             False    ContainersReady   False    PodScheduled      True  Volumes:   default-token-s7jhb:     Type:        Secret (a volume populated by a Secret)     SecretName:  default-token-s7jhb     Optional:    false QoS Class:       BestEffort Node-Selectors:   Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s                  node.kubernetes.io/unreachable:NoExecute for 300s Events:   Type     Reason     Age                   From                 Message   ----     ------     ----                  ----                 -------   Normal   Scheduled  17m                   default-scheduler    Successfully assigned default/httpd-deployment-5fd659bb4d-wsjfm to kubeworker   Normal   Pulling    16m (x4 over 17m)     kubelet, kubeworker  Pulling image  "192.168.56.100:5000/middleware/httpd/httpd_custom_image:2.4.41"   Warning  Failed     16m (x4 over 17m)     kubelet, kubeworker Failed to pull image "192.168.56.100:5000/middleware/httpd/httpd_custom_image:2.4.41":  rpc error: code = Unknown desc = Error response from daemon:  Get http://192.168.56.100:5000/v2/middleware/httpd/httpd_custom_image/manifests/2.4.41: no basic auth credentials   Warning  Failed     16m (x4 over 17m)     kubelet, kubeworker  Error: ErrImagePull   Normal   BackOff    16m (x6 over 17m)     kubelet, kubeworker  Back-off pulling image "192.168.56.100:5000/middleware/httpd/httpd_custom_image:2.4.41"   Warning  Failed     2m50s (x65 over 17m)  kubelet, kubeworker  Error: ImagePullBackOff [root@nrson kubernetes_deploy]#

pods의 describe 정보를 확인해 보니 다음과 같은 로그를 확인할 수 있습니다.

Events:    Type     Reason     Age                   From                 Message    ----     ------     ----                  ----                 -------    Normal   Scheduled  17m                    default-scheduler    Successfully assigned default/httpd-deployment-5fd659bb4d-wsjfm to kubeworker    Normal   Pulling    16m (x4 over 17m)      kubelet, kubeworker  Pulling image "192.168.56.100:5000/middleware/httpd/httpd_custom_image:2.4.41"    Warning  Failed     16m (x4 over 17m)     kubelet, kubeworker  Failed to pull image  "192.168.56.100:5000/middleware/httpd/httpd_custom_image:2.4.41":  rpc error: code = Unknown desc = Error response from daemon:  Get http://192.168.56.100:5000/v2/middleware/httpd/httpd_custom_image/manifests/2.4.41: no basic auth credentials    Warning  Failed     16m (x4 over 17m)     kubelet, kubeworker  Error: ErrImagePull    Normal   BackOff    16m (x6 over 17m)     kubelet, kubeworker  Back-off pulling image "192.168.56.100:5000/middleware/httpd/httpd_custom_image:2.4.41"    Warning  Failed     2m50s (x65 over 17m)  kubelet, kubeworker  Error: ImagePullBackOff

해당 로그는 Kubernetes에서 Docker Private Registry를 통해 이미지를 Pull 할 때 Registry에 접근할 수 있는 권한이 없어 발생하는 문제입니다.

3. 문제 해결

1) Kubernetes secret 활용

위 문제를 해결하기 위해서는 Kubernetes의 secret을 등록하여 해당 namespace에 권한을 부여해야 합니다.

먼저 Docker Private Registry를 사용하기 위한 secret을 생성해 보도록 하겠습니다.

[root@nrson kubernetes_deploy]# kubectl create secret docker-registry docker-registry-login \ > --docker-server=192.168.56.100:5000 \ > --docker-username=admin \ > --docker-password=admin123 \ > --docker-email=son.nara@lgcns.com \ > --namespace=default  secret/docker-registry-login created [root@nrson kubernetes_deploy]# kubectl get secrets NAME                    TYPE                                  DATA   AGE default-token-s7jhb     kubernetes.io/service-account-token   3      9h docker-registry-login   kubernetes.io/dockerconfigjson        1      67m [root@nrson kubernetes_deploy]#

위와 같이 secret이 생성되면 deployment에 해당 정보를 적용하도록 합니다.

기존에 생성한 deployment.yaml 파일을 아래와 같이 수정합니다.

apiVersion: apps/v1 kind: Deployment metadata:   name: httpd-deployment   labels:     app: httpd spec:   replicas: 1   selector:     matchLabels:       app: httpd   template:     metadata:       labels:         app: httpd     spec:       containers:       - name: httpd         image: 192.168.56.100:5000/middleware/httpd/httpd_custom_image:2.4.41         imagePullPolicy: Always         ports:         - containerPort: 80       imagePullSecrets:         - name: docker-registry-login --- apiVersion: v1 kind: Service metadata:   name: httpd spec:   selector:     app: httpd   ports:     - protocol: TCP       port: 80       targetPort: 80   type: ClusterIP --- apiVersion: extensions/v1beta1 kind: Ingress metadata:   name: httpd-ingress   namespace: default spec:   rules:   - host: test.nrson.co.kr     http:       paths:       - backend:           serviceName: httpd           servicePort: 80         path: /

위와 같이 imagePullSecrets를 추가하여 생성한 secret이 해당 deployment에 적용되도록 반영합니다.

이후 kubectl apply 명령어로 deployment를 갱신합니다.

[root@nrson kubernetes_deploy]# kubectl apply -f deployment.yaml  Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply deployment.apps/httpd-deployment configured Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply service/httpd configured Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply ingress.extensions/httpd-ingress configured [root@nrson kubernetes_deploy]# kubectl get pods --all-namespaces -o wide NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE    READINESS GATES default       httpd-deployment-7696db7c86-fmwjq         1/1     Running   0          27s   10.233.104.13    kubeworker               kube-system   calico-kube-controllers-cbbd89c57-6ffnw   1/1     Running   1          9h    192.168.56.102   kubeworker               kube-system   calico-node-cpcgj                         1/1     Running   3          9h    192.168.56.102   kubeworker               kube-system   calico-node-ltwdb                         1/1     Running   5          9h    192.168.56.101   kubemaster               kube-system   coredns-58687784f9-md5rs                  1/1     Running   1          9h    10.233.116.3     kubemaster               kube-system   coredns-58687784f9-mjrkg                  1/1     Running   1          9h    10.233.104.5     kubeworker               kube-system   dns-autoscaler-79599df498-4vgk2           1/1     Running   1          9h    10.233.116.4     kubemaster               kube-system   kube-apiserver-kubemaster                 1/1     Running   3          9h    192.168.56.101   kubemaster               kube-system   kube-controller-manager-kubemaster        1/1     Running   1          9h    192.168.56.101   kubemaster               kube-system   kube-proxy-5tkkz                          1/1     Running   1          9h    192.168.56.102   kubeworker               kube-system   kube-proxy-jlxhd                          1/1     Running   3          9h    192.168.56.101   kubemaster               kube-system   kube-scheduler-kubemaster                 1/1     Running   1          9h    192.168.56.101   kubemaster               kube-system   kubernetes-dashboard-556b9ff8f8-w8cfq     1/1     Running   2          9h    10.233.104.4     kubeworker               kube-system   nginx-proxy-kubeworker                    1/1     Running   1          9h    192.168.56.102   kubeworker               kube-system   nodelocaldns-4wmkz                        1/1     Running   1          9h    192.168.56.101   kubemaster               kube-system   nodelocaldns-vvkjj                        1/1     Running   1          9h    192.168.56.102   kubeworker               [root@nrson kubernetes_deploy]#

위와 같이 imagePullBackOff 상태였던 httpd-deployment가 Running 상태로 변경된 것을 확인할 수 있습니다.

2) $HOME/.docker/config.json 활용

연결하고자 하는 docker repository에 사전에 로그인을 수행합니다.

[root@kubemaster .docker]# docker login http://192.168.56.100:5000 Username: admin Password:  WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded [root@kubemaster .docker]# pwd /root/.docker [root@kubemaster .docker]# cat config.json {         "auths": {                 "192.168.56.100:5000": {                         "auth": "YWRtaW46YWRtaW4xMjM="                 }         },         "HttpHeaders": {                 "User-Agent": "Docker-Client/19.03.5 (linux)"         } }[root@kubemaster .docker]#

위와 같이 docker에 로그인을 진행하면 /root/.docker/config.json 파일에 위와 같이 auth key가 자동으로 등록됩니다. 이후에는 별도로 Kubernetes Secret을 등록하지 않아도 Docker image를 가져올 수 있습니다.

그럼 마지막으로 httpd가 정상 구동되었는지 여부를 확인해 보도록 하겠습니다.

[root@nrson kubernetes_deploy]# kubectl get service -o wide NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   SELECTOR httpd        ClusterIP   10.233.19.128           80/TCP    44m   app=httpd kubernetes   ClusterIP   10.233.0.1              443/TCP   9h     [root@nrson kubernetes_deploy]#

현재 사용 중인 ClusterIP 방식으로 호출을 시도해 보겠습니다.

80 포트로 접근하면 아래와 같이 It works! 텍스트가 출력됩니다.

[root@nrson kubernetes_deploy]# curl http://10.233.19.128 It works! [root@nrson kubernetes_deploy]#

위와 같이 Kubernetes에서 Docker Private Registry에 접근하기 위한 secret을 추가하고 deployment에 반영하면, Docker Image를 정상적으로 Pull 하는 것을 확인할 수 있습니다.