Kubernetes - etcdmain: open /etc/kubernetes/pki/etcd/peer.crt: permission denied

etcdmain: open /etc/kubernetes/pki/etcd/peer.crt: permission denied 에러 발생 시 대응 방법

1) Kubernetes 기동 시점에 etcd Process가 기동되지 않고 Restart 되는 현상

[root@kubemaster ~]# docker ps -a 
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                      PORTS               NAMES 
1757b7980995        68c3eb07bfc3           "kube-apiserver --..."   21 seconds ago      Exited (1) 21 seconds ago                       k8s_kube-apiserver_kube-apiserver-kubemaster_kube-system_4c06c3fb6b086d48ae106fc19638ee2a_7 
3efa6966367b        d75082f1d121           "kube-controller-m..."   29 seconds ago      Exited (1) 28 seconds ago                       k8s_kube-controller-manager_kube-controller-manager-kubemaster_kube-system_d3aba460bc6511cf3a52d2be65fd4bab_7 
1d831a269eed        2c4adeb21b4f           "etcd --advertise-..."   35 seconds ago      Exited (1) 35 seconds ago                       k8s_etcd_etcd-kubemaster_kube-system_4c3bc8596abc23ae3e13b3d4e3de0683_7 
424a3b751ec3        b0b3c4c404da           "kube-scheduler --..."   11 minutes ago      Up 11 minutes                                   k8s_kube-scheduler_kube-scheduler-kubemaster_kube-system_ecae9d12d3610192347be3d1aa5aa552_0 
c138cb03f1fa        k8s.gcr.io/pause:3.1   "/pause"                 11 minutes ago      Up 11 minutes                                   k8s_POD_kube-controller-manager-kubemaster_kube-system_d3aba460bc6511cf3a52d2be65fd4bab_0 
dba3d7106c1b        k8s.gcr.io/pause:3.1   "/pause"                 11 minutes ago      Up 11 minutes                                   k8s_POD_etcd-kubemaster_kube-system_4c3bc8596abc23ae3e13b3d4e3de0683_0 
b83da93dd8cc        k8s.gcr.io/pause:3.1   "/pause"                 11 minutes ago      Up 11 minutes                                   k8s_POD_kube-scheduler-kubemaster_kube-system_ecae9d12d3610192347be3d1aa5aa552_0 
cde4243d2985        k8s.gcr.io/pause:3.1   "/pause"                 11 minutes ago      Up 11 minutes                                   k8s_POD_kube-apiserver-kubemaster_kube-system_4c06c3fb6b086d48ae106fc19638ee2a_0 
[root@kubemaster ~]# 

2) Docker log 확인

[root@kubemaster ~]# docker logs 1d831a269eed 
2019-08-17 13:39:48.422847 I | etcdmain: etcd Version: 3.3.10 
2019-08-17 13:39:48.423041 I | etcdmain: Git SHA: 27fc7e2 
2019-08-17 13:39:48.423044 I | etcdmain: Go Version: go1.10.4 
2019-08-17 13:39:48.423046 I | etcdmain: Go OS/Arch: linux/amd64 
2019-08-17 13:39:48.423049 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2 
2019-08-17 13:39:48.423250 I | embed: peerTLS: cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, ca = , trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file =  
2019-08-17 13:39:48.423592 C | etcdmain: open /etc/kubernetes/pki/etcd/peer.crt: permission denied 
[root@kubemaster ~]#

3) 진단

로그 확인 결과 etcdmain: open /etc/kubernetes/pki/etcd/peer.crt: permission denied 에러 발생 확인

 

4) 대응

Selinux가 Enforcing일 경우 접근 제한이 걸리므로 getenforce를 확인하고 Enforcing일 경우 Permissive 또는 Disabled로 변경함

[root@kubemaster ~]# getenforce
Enforcing
[root@kubemaster ~]#

변경은

[root@kubemaster ~]# vi /etc/selinux/config  

# This file controls the state of SELinux on the system. 
# SELINUX= can take one of these three values: 
#     enforcing - SELinux security policy is enforced. 
#     permissive - SELinux prints warnings instead of enforcing. 
#     disabled - No SELinux policy is loaded. 
#SELINUX=enforcing 
SELINUX=disabled 
# SELINUXTYPE= can take one of three values: 
#     targeted - Targeted processes are protected, 
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection. 
SELINUXTYPE=targeted

위와 같이 변경하거나 임시로 setenforce 0으로 해당 콘솔에만 적용한다.

[root@kubemaster ~]# setenforce 0
[root@kubemaster ~]# getenforce
Permissive
[root@kubemaster ~]#

변경이 확인되었으면 재기동을 수행한다.