티스토리 뷰

728x90
반응형

이번 포스팅에서는 Kubernetes의 Pod들이 배치될 Worker 노드를 설치하고 Master Node와 연결하는 과정을 살펴보겠습니다.

지난 Kubernetes 가이드는 아래를 참고하세요.

 

[Container Management] Kubernetes Master Node 설치

[Container Management] Kubernetes Dashboard Install & Setting

Kubernetes Worker Node 설치

1. firewall-cmd를 활용하여 오픈할 방화벽 port를 정의합니다.

firewall-cmd --zone=public --permanent --add-port=10250/tcp

firewall-cmd --zone=public --permanent --add-port=30000-32767/tcp

firewall-cmd --reload

Worker Node에서 사용하는 포트는 다음과 같습니다.

Protocol

Direction

Port Range

Purpose

Used By

TCP

Inbound

10250

Kubelet API

Self, Control plane

TCP

Inbound

30000 - 32767

NodePort Services**

All

2. Docker 및 Kubernetes 설치

[root@kubeworker ~]# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
[root@kubeworker ~]#

kubernetes repository를 추가하여 kubeadm, kubectl, kubelet을 설치합니다.

[root@kubeworker ~]# yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.navercorp.com
 * extras: mirror.navercorp.com
 * updates: mirror.navercorp.com
Resolving Dependencies
--> Running transaction check
---> Package kubeadm.x86_64 0:1.15.1-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.15.1-0.x86_64
--> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.15.1-0.x86_64
---> Package kubectl.x86_64 0:1.15.1-0 will be installed
---> Package kubelet.x86_64 0:1.15.1-0 will be installed
--> Processing Dependency: socat for package: kubelet-1.15.1-0.x86_64
--> Processing Dependency: conntrack for package: kubelet-1.15.1-0.x86_64
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-4.el7 will be installed
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-4.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-4.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-4.el7.x86_64
--> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-4.el7.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-4.el7.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-4.el7.x86_64
---> Package cri-tools.x86_64 0:1.13.0-0 will be installed
---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed
---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed
--> Running transaction check
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-9.el7 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================================================================================
 Package                                                           Arch                                              Version                                                    Repository                                             Size
============================================================================================================================================================================================================================================
Installing:
 kubeadm                                                           x86_64                                            1.15.1-0                                                   kubernetes                                            8.9 M
 kubectl                                                           x86_64                                            1.15.1-0                                                   kubernetes                                            9.5 M
 kubelet                                                           x86_64                                            1.15.1-0                                                   kubernetes                                             22 M
Installing for dependencies:
 conntrack-tools                                                   x86_64                                            1.4.4-4.el7                                                base                                                  186 k
 cri-tools                                                         x86_64                                            1.13.0-0                                                   kubernetes                                            5.1 M
 kubernetes-cni                                                    x86_64                                            0.7.5-0                                                    kubernetes                                             10 M
 libnetfilter_cthelper                                             x86_64                                            1.0.0-9.el7                                                base                                                   18 k
 libnetfilter_cttimeout                                            x86_64                                            1.0.0-6.el7                                                base                                                   18 k
 libnetfilter_queue                                                x86_64                                            1.0.2-2.el7_2                                              base                                                   23 k
 socat                                                             x86_64                                            1.7.3.2-2.el7                                              base                                                  290 k

Transaction Summary
============================================================================================================================================================================================================================================
Install  3 Packages (+7 Dependent packages)

Total download size: 56 M
Installed size: 251 M
Downloading packages:
(1/10): conntrack-tools-1.4.4-4.el7.x86_64.rpm                                                                                                                                                                       | 186 kB  00:00:00     
warning: /var/cache/yum/x86_64/7/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY  0.0 B/s | 186 kB  --:--:-- ETA 
Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed
(2/10): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm                                                                                                               | 5.1 MB  00:00:01     
(3/10): aa386b8f2cac67415283227ccb01dc043d718aec142e32e1a2ba6dbd5173317b-kubeadm-1.15.1-0.x86_64.rpm                                                                                                                 | 8.9 MB  00:00:02     
(4/10): f27b0d7e1770ae83c9fce9ab30a5a7eba4453727cdc53ee96dc4542c8577a464-kubectl-1.15.1-0.x86_64.rpm                                                                                                                 | 9.5 MB  00:00:01     
(5/10): libnetfilter_cthelper-1.0.0-9.el7.x86_64.rpm                                                                                                                                                                 |  18 kB  00:00:00     
(6/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm                                                                                                                                                                  |  23 kB  00:00:00     
(7/10): socat-1.7.3.2-2.el7.x86_64.rpm                                                                                                                                                                               | 290 kB  00:00:00     
(8/10): libnetfilter_cttimeout-1.0.0-6.el7.x86_64.rpm                                                                                                                                                                |  18 kB  00:00:00     
(9/10): 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm                                                                                                           |  10 MB  00:00:01     
(10/10): f5edc025972c2d092ac41b05877c89b50cedaa7177978d9e5e49b5a2979dbc85-kubelet-1.15.1-0.x86_64.rpm                                                                                                                |  22 MB  00:00:03     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                       8.7 MB/s |  56 MB  00:00:06     
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0xA7317B0F:
 Userid     : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
 Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Importing GPG key 0x3E1BA8D5:
 Userid     : "Google Cloud Packages RPM Signing Key <gc-team@google.com>"
 Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5
 From       : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : socat-1.7.3.2-2.el7.x86_64                                                                                                                                                                                              1/10 
  Installing : libnetfilter_cthelper-1.0.0-9.el7.x86_64                                                                                                                                                                                2/10 
  Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64                                                                                                                                                                                 3/10 
  Installing : cri-tools-1.13.0-0.x86_64                                                                                                                                                                                               4/10 
  Installing : libnetfilter_cttimeout-1.0.0-6.el7.x86_64                                                                                                                                                                               5/10 
  Installing : conntrack-tools-1.4.4-4.el7.x86_64                                                                                                                                                                                      6/10 
  Installing : kubernetes-cni-0.7.5-0.x86_64                                                                                                                                                                                           7/10 
  Installing : kubelet-1.15.1-0.x86_64                                                                                                                                                                                                 8/10 
  Installing : kubectl-1.15.1-0.x86_64                                                                                                                                                                                                 9/10 
  Installing : kubeadm-1.15.1-0.x86_64                                                                                                                                                                                                10/10 
  Verifying  : kubectl-1.15.1-0.x86_64                                                                                                                                                                                                 1/10 
  Verifying  : libnetfilter_cttimeout-1.0.0-6.el7.x86_64                                                                                                                                                                               2/10 
  Verifying  : cri-tools-1.13.0-0.x86_64                                                                                                                                                                                               3/10 
  Verifying  : libnetfilter_queue-1.0.2-2.el7_2.x86_64                                                                                                                                                                                 4/10 
  Verifying  : libnetfilter_cthelper-1.0.0-9.el7.x86_64                                                                                                                                                                                5/10 
  Verifying  : kubelet-1.15.1-0.x86_64                                                                                                                                                                                                 6/10 
  Verifying  : kubeadm-1.15.1-0.x86_64                                                                                                                                                                                                 7/10 
  Verifying  : kubernetes-cni-0.7.5-0.x86_64                                                                                                                                                                                           8/10 
  Verifying  : socat-1.7.3.2-2.el7.x86_64                                                                                                                                                                                              9/10 
  Verifying  : conntrack-tools-1.4.4-4.el7.x86_64                                                                                                                                                                                     10/10 

Installed:
  kubeadm.x86_64 0:1.15.1-0                                                     kubectl.x86_64 0:1.15.1-0                                                     kubelet.x86_64 0:1.15.1-0                                                    

Dependency Installed:
  conntrack-tools.x86_64 0:1.4.4-4.el7  cri-tools.x86_64 0:1.13.0-0  kubernetes-cni.x86_64 0:0.7.5-0  libnetfilter_cthelper.x86_64 0:1.0.0-9.el7  libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7  libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 
  socat.x86_64 0:1.7.3.2-2.el7         

Complete!
[root@kubeworker ~]#

Kubernetes Master Node 연동

1. swapoff -a

[root@kubeworker ~]# swapoff -a
[root@kubeworker ~]#

2. kubeadm join

연동 방법은 간단합니다. 기존 Master Node에서 kubeadm init 시 생성된 token 정보를 기반으로 join을 시도합니다.


[root@guruson ~]# kubeadm init 
[init] Using Kubernetes version: v1.15.1 
[preflight] Running pre-flight checks 
[preflight] Pulling images required for setting up a Kubernetes cluster 
[preflight] This might take a minute or two, depending on the speed of your internet connection 
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull' 
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml
[kubelet-start] Activating the kubelet service 
[certs] Using certificateDir folder "/etc/kubernetes/pki" 
[certs] Generating "front-proxy-ca" certificate and key 
[certs] Generating "front-proxy-client" certificate and key 
[certs] Generating "etcd/ca" certificate and key 
[certs] Generating "etcd/peer" certificate and key 
[certs] etcd/peer serving cert is signed for DNS names [guruson localhost] and IPs [222.234.124.110 127.0.0.1 ::1] 
[certs] Generating "etcd/healthcheck-client" certificate and key 
[certs] Generating "etcd/server" certificate and key 
[certs] etcd/server serving cert is signed for DNS names [guruson localhost] and IPs [222.234.124.110 127.0.0.1 ::1] 
[certs] Generating "apiserver-etcd-client" certificate and key 
[certs] Generating "ca" certificate and key 
[certs] Generating "apiserver" certificate and key 
[certs] apiserver serving cert is signed for DNS names [guruson kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 222.234.124.110] 
[certs] Generating "apiserver-kubelet-client" certificate and key 
[certs] Generating "sa" key and public key 
[kubeconfig] Using kubeconfig folder "/etc/kubernetes" 
[kubeconfig] Writing "admin.conf" kubeconfig file 
[kubeconfig] Writing "kubelet.conf" kubeconfig file 
[kubeconfig] Writing "controller-manager.conf" kubeconfig file 
[kubeconfig] Writing "scheduler.conf" kubeconfig file 
[control-plane] Using manifest folder "/etc/kubernetes/manifests" 
[control-plane] Creating static Pod manifest for "kube-apiserver" 
[control-plane] Creating static Pod manifest for "kube-controller-manager" 
[control-plane] Creating static Pod manifest for "kube-scheduler" 
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" 
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s 
[kubelet-check] Initial timeout of 40s passed. 
[apiclient] All control plane components are healthy after 45.506400 seconds 
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace 
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster 
[upload-certs] Skipping phase. Please see --upload-certs 
[mark-control-plane] Marking the node guruson as control-plane by adding the label "node-role.kubernetes.io/master=''
[mark-control-plane] Marking the node guruson as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] 
[bootstrap-token] Using token: deb19a.7yfa212rg0exg0c9 
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles 
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials 
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token 
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster 
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace 
[addons] Applied essential addon: CoreDNS 
[addons] Applied essential addon: kube-proxy 

Your Kubernetes control-plane has initialized successfully! 

To start using your cluster, you need to run the following as a regular user: 

  mkdir -p $HOME/.kube 
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 
  sudo chown $(id -u):$(id -g) $HOME/.kube/config 

You should now deploy a pod network to the cluster. 
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: 
  https://kubernetes.io/docs/concepts/cluster-administration/addons/ 

Then you can join any number of worker nodes by running the following on each as root: 

kubeadm join 222.234.124.110:6443 --token deb19a.7yfa212rg0exg0c9 \ 
    --discovery-token-ca-cert-hash sha256:20d38dd05c158fe88fecd1b219ba9a5e02e5ea66ad612b404678571d104754c3  
[root@guruson ~]#


혹시나 키 값을 분실하였거나 Expired 되었을 경우 다음과 같이 확인 또는 재 생성이 가능합니다.

a) master node token 값 확인 (master node에서 명령어 실행)

[root@guruson ~]# kubeadm token list
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
y0ilnu.ospwoi1kbxscy9iz   13h       2019-08-04T20:29:54+09:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token
[root@guruson ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
325d28e81247bea994ebf2fa6bccb5af211aa3e6a2a762af800992b0080fb103
[root@guruson ~]#

위와 같이 token을 확인하고 Expired 이전 이라면 해당 그대로 사용이 가능합니다.

적용 방법은 kubeadm join --token [TOKEN] --discovery-token-ca-cert-hash sha256:[OPENSSL_ENCRYPT] [MASTER_APISERVER_IP]:6443

[root@guruson ~]# kubeadm join --token y0ilnu.ospwoi1kbxscy9iz --discovery-token-ca-cert-hash sha256:325d28e81247bea994ebf2fa6bccb5af211aa3e6a2a762af800992b0080fb103 ^C
[root@guruson ~]#

와 같습니다.

b) master node token이 만료되었을 경우 (master node에서 명령어 실행)

[root@guruson ~]# kubeadm token create
bzgy3x.ltfohixm3ijxm67l
[root@guruson ~]# kubeadm token list
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
bzgy3x.ltfohixm3ijxm67l   23h       2019-08-05T08:15:29+09:00   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token
y0ilnu.ospwoi1kbxscy9iz   12h       2019-08-04T20:29:54+09:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token
[root@guruson ~]#

위와 같이 신규로 생성이 가능합니다.

3. Master Node 연결 상태 확인

Master Node와 Workder Node의 연결상태를 확인합니다.

[root@guruson ~]# kubectl get nodes
NAME         STATUS    ROLES     AGE       VERSION
guruson      Ready     master    11h       v1.15.1
kubeworker   Ready     <none>    3m18s     v1.15.1
[root@guruson ~]# 

연결이 정상적으로 Ready 된 것을 확인할 수 있습니다.

4. Workder Node 상태 확인

[root@kubeworker ~]# docker ps -a
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS               NAMES
703c6923dadc        weaveworks/weave-npc    "/usr/bin/weave-npc"     53 minutes ago      Up 53 minutes                           k8s_weave-npc_weave-net-prcb5_kube-system_35d5017f-b830-48d7-84df-3a40b8aebd5a_0
f9f84fe7f91e        weaveworks/weave-kube   "/home/weave/launch.   53 minutes ago      Up 53 minutes                           k8s_weave_weave-net-prcb5_kube-system_35d5017f-b830-48d7-84df-3a40b8aebd5a_0
45ae9c7dbfcc        k8s.gcr.io/kube-proxy   "/usr/local/bin/kube   53 minutes ago      Up 53 minutes                           k8s_kube-proxy_kube-proxy-6bgfw_kube-system_d70f96e3-92ac-483a-96f0-209bedd1bfad_0
1cf6f307fee1        k8s.gcr.io/pause:3.1    "/pause"                 53 minutes ago      Up 53 minutes                           k8s_POD_weave-net-prcb5_kube-system_35d5017f-b830-48d7-84df-3a40b8aebd5a_0
fae7fcda7abc        k8s.gcr.io/pause:3.1    "/pause"                 53 minutes ago      Up 53 minutes                           k8s_POD_kube-proxy-6bgfw_kube-system_d70f96e3-92ac-483a-96f0-209bedd1bfad_0
[root@kubeworker ~]# docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy   v1.15.1             89a062da739d        2 weeks ago         82.4MB
weaveworks/weave-kube   2.5.2               f04a043bb67a        2 months ago        148MB
weaveworks/weave-npc    2.5.2               5ce48e0d813c        2 months ago        49.6MB
k8s.gcr.io/pause        3.1                 da86e6ba6ca1        19 months ago       742kB
[root@kubeworker ~]# 

Workder Node는 kube-proxy와 weaveworks Docker 이미지가 기동되어 있습니다.

 

이번 포스팅에서는 Master Node와 Worker Node를 연동하는 방법에 대해 살펴보았습니다.

다음 포스팅에서는 실제 Pod를 생성하여 애플리케이션을 디플로이하기 위한 구조를 생성해 보도록 하겠습니다.

728x90
반응형