티스토리 뷰

728x90
반응형

지금부터 살펴볼 내용은 어쩌면 지금까지 살펴본 다양한 Kubernetes 포스팅의 완결판이 될 수도 있을 것 같습니다.

지금까지 다양한 Component를 살펴보며 Kubernetes가 제공하는 Request Flow에 대해 알아보고 구성해 보았습니다.

다만 각 컴포넌트들을 직접 구성하기에는 자세히 설명을 하였지만 처음 접하는 분들에게는 어려운 과정이 될 것입니다.

이에 누구나 손쉽게 따라 구성해 볼 수 있게 Shell Script와 YAML 파일을 작성해 보았습니다.

이전 포스팅은 아래를 참고해 주시기 바랍니다.

 

[Container Management] Kubernetes Master Node 설치

[Container Management] Kubernetes Dashboard Install & Setting

[Container Management] Kubernetes Woker Node Install & Setting

[Container Management] Kubernetes Pod 생성 가이드

[Container Management] Kubernetes Service 생성 가이드

[Container Management] Kubernetes 애플리케이션 관리

[Container Management] Kubernetes ingress & ingress controller

 

특히 지난 Part 1. Install에서 살펴본 OneStep Install 과정은 다음을 참고해 주세요.

[Container Management] Kubernetes OneStep Install & Boot & Setting (Part 1. Install)

Kubernetes One Step Boot & Setting

이번 포스팅에서는 지난 시간에 과제로 드린 Master Node & Worker Node가 동일하게 구성되어 있다는 가정하에 진행하도록 하겠습니다.

혹시나 내용을 스킵하고 빠르게 수행해 보고 싶으신 분은 제일 하단으로 이동하시기 바랍니다.

1. Git Repository Clone

먼저 진행 할 작업은 사전에 만들어 둔 github repository에 있는 소스를 다운로드 받습니다.

앞선 포스팅에서 이미 git clone을 받으신 분은 넘어가셔도 됩니다.

- git install

[root@kubemaster ~]# yum install -y git
Last login: Sat Aug 17 17:13:52 2019
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: data.aonenetworks.kr
 * extras: data.aonenetworks.kr
 * updates: data.aonenetworks.kr
base                                                                                                                                                                                                                 | 3.6 kB  00:00:00     
extras                                                                                                                                                                                                               | 3.4 kB  00:00:00     
updates                                                                                                                                                                                                              | 3.4 kB  00:00:00     
(1/4): base/7/x86_64/primary_db                                                                                                                                                                                      | 6.0 MB  00:00:00     
(2/4): base/7/x86_64/group_gz                                                                                                                                                                                        | 166 kB  00:00:03     
(3/4): extras/7/x86_64/primary_db                                                                                                                                                                                    | 215 kB  00:00:04 
...
...
...
Installed:
  git.x86_64 0:1.8.3.1-20.el7                                                                                                                                                                                                               

Dependency Installed:
  perl.x86_64 4:5.16.3-294.el7_6                 perl-Carp.noarch 0:1.26-244.el7                    perl-Encode.x86_64 0:2.51-7.el7                perl-Error.noarch 1:0.17020-2.el7          perl-Exporter.noarch 0:5.68-3.el7            
  perl-File-Path.noarch 0:2.09-2.el7             perl-File-Temp.noarch 0:0.23.01-3.el7              perl-Filter.x86_64 0:1.49-3.el7                perl-Getopt-Long.noarch 0:2.40-3.el7       perl-Git.noarch 0:1.8.3.1-20.el7             
  perl-HTTP-Tiny.noarch 0:0.033-3.el7            perl-PathTools.x86_64 0:3.40-5.el7                 perl-Pod-Escapes.noarch 1:1.04-294.el7_6       perl-Pod-Perldoc.noarch 0:3.20-4.el7       perl-Pod-Simple.noarch 1:3.28-4.el7          
  perl-Pod-Usage.noarch 0:1.63-3.el7             perl-Scalar-List-Utils.x86_64 0:1.27-248.el7       perl-Socket.x86_64 0:2.010-4.el7               perl-Storable.x86_64 0:2.45-3.el7          perl-TermReadKey.x86_64 0:2.30-20.el7        
  perl-Text-ParseWords.noarch 0:3.29-4.el7       perl-Time-HiRes.x86_64 4:1.9725-3.el7              perl-Time-Local.noarch 0:1.2300-2.el7          perl-constant.noarch 0:1.27-2.el7          perl-libs.x86_64 4:5.16.3-294.el7_6          
  perl-macros.x86_64 4:5.16.3-294.el7_6          perl-parent.noarch 1:0.225-244.el7                 perl-podlators.noarch 0:2.5.1-3.el7            perl-threads.x86_64 0:1.87-4.el7           perl-threads-shared.x86_64 0:1.43-6.el7      
  rsync.x86_64 0:3.1.2-6.el7_6.1                

Complete!
[root@kubemaster ~]#

- git clone

[root@kubemaster ~]# mkdir git_repo
[root@kubemaster ~]# cd git_repo/
[root@kubemaster git_repo]# ls
[root@kubemaster git_repo]# git clone https://github.com/sonnaraon/kubernetesonestepboot.git
Cloning into 'kubernetesonestepboot'...
remote: Enumerating objects: 149, done.
remote: Counting objects: 100% (149/149), done.
remote: Compressing objects: 100% (89/89), done.
remote: Total 149 (delta 57), reused 146 (delta 54), pack-reused 0
Receiving objects: 100% (149/149), 32.82 KiB | 0 bytes/s, done.
Resolving deltas: 100% (57/57), done.
[root@kubemaster git_repo]# ls -la
drwxr-xr-x. 3 root root  35  817 11:18 .
dr-xr-x---. 4 root root 142  817 11:18 ..
drwxr-xr-x. 5 root root 100  817 11:18 kubernetesonestepboot
[root@kubemaster git_repo]# cd kubernetesonestepboot/
[root@kubemaster kubernetesonestepboot]# ls
ReadMe.adoc  kubernetes_onestepboot  kubernetes_onestepinstall
[root@kubemaster kubernetesonestepboot]# ls -la
drwxr-xr-x. 5 root root  100  817 11:18 .
drwxr-xr-x. 3 root root   35  817 11:18 ..
drwxr-xr-x. 8 root root  163  817 11:18 .git
-rw-r--r--. 1 root root 4212  817 11:18 ReadMe.adoc
drwxr-xr-x. 2 root root 4096  817 11:18 kubernetes_onestepboot
drwxr-xr-x. 2 root root   95  817 11:18 kubernetes_onestepinstall
[root@kubemaster kubernetesonestepboot]#

아래서 설명해 드리겠지만, 다운로드 받은 git repository kubernetesonestepboot project는

1) kubernetes_onestepinstall : kubernetes 설치용 Shell Script

2) kubernetes_onestepboot : kubernetes 기동 및 기본 환경 구성용 Shell Script

로 구성되어 있습니다.

kubernetes_onestepboot 디렉토리 하위로 이동합니다.

[root@kubemaster kubernetes_onestepboot]# ls -la
total 104
drwxr-xr-x. 2 root root  4096 Aug 17 17:20 .
drwxr-xr-x. 5 root root   100 Aug 17 17:20 ..
-rwxr-xr-x. 1 root root 11175 Aug 17 17:20 0.kubernetesOneStepBoot.sh
-rwxr-xr-x. 1 root root    69 Aug 17 17:20 0.kubernetesOneStepDown.sh
-rwxr-xr-x. 1 root root  1213 Aug 17 17:20 0.kubernetesOneStepInfo.sh
-rw-r--r--. 1 root root   142 Aug 17 17:20 1.kubeenv.sh
-rw-r--r--. 1 root root   419 Aug 17 17:20 2.kubectlapply.sh
-rw-r--r--. 1 root root   540 Aug 17 17:20 3.securitykey.sh
-rw-r--r--. 1 root root   413 Aug 17 17:20 4.encrypt.sh
-rw-r--r--. 1 root root   748 Aug 17 17:20 5.new-apache-docker.yaml
-rw-r--r--. 1 root root   674 Aug 17 17:20 5.new-apache-docker.yaml_org
-rw-r--r--. 1 root root   217 Aug 17 17:20 6.new-internal-service.yaml
-rw-r--r--. 1 root root  6067 Aug 17 17:20 7.new-mandatory.yaml
-rw-r--r--. 1 root root  6045 Aug 17 17:20 7.new-mandatory.yaml_org
-rw-r--r--. 1 root root   357 Aug 17 17:20 8.new-ingress.yaml
-rw-r--r--. 1 root root   339 Aug 17 17:20 9-1.new-ingress-svc-externalip.yaml
-rw-r--r--. 1 root root   306 Aug 17 17:20 9-2.ingress-svc-nodeport.yaml
-rw-r--r--. 1 root root  1143 Aug 17 17:20 9-3.default-backend.yaml
-rw-r--r--. 1 root root  3230 Aug 17 17:20 kubernetes_masternode_setting.txt
-rw-r--r--. 1 root root  6784 Aug 17 17:20 net.yaml
-rw-r--r--. 1 root root  6928 Aug 17 17:20 recommended.yaml
[root@kubemaster kubernetes_onestepboot]#

install 시에는 0.kubernetesOneStepInstall.sh (or 1.kubernetesOneStepInstallByInternet.sh) 하나의 실행 파일이 존재하였지만, boot Script에는 총 3개의 실행 파일이 존재합니다.


- 0.kubernetesOneStepBoot.sh : Kubernetes Master Node, Worker Node 기동, Component 생성 및 샘플 애플리케이션 생성 등

- 0.kubernetesOneStepDown.sh : Kubernetes Master Node, Worker Node 다운

- 0.kubernetesOneStepInfo.sh : Kubernetes 기동 이후 각종 정보 확인


2. Kubernetes 0.kubernetesOneStepBoot.sh로 구동 및 설정하기

먼저 살펴볼 정보는 0.kubernetesOneStepBoot.sh입니다.

#!/bin/bash
StartTime=$(date +%s)
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "********************************************* Kubernetes Master Node 구성을 시작합니다. *************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"

echo "#####"
echo "##### Set ENV"
echo "#####"
export MASTER_NODE=192.168.xxx.xxx
export WORKER_NODE=192.168.xxx.xxx
export WORKER_PASS=xxxxxx
export DOCKER_REGISTRY="192.168.xxx.xxx:port"

echo "#####"
echo "##### Check the firewall"
echo "#####"
count=$(systemctl status firewalld | grep dead | wc -l)
if(( "${count}" == 0 )); 
then
        echo "firewalld가 기동되어 있습니다."
        echo "firewalld를 다운 하시겠습니까? (y/n)"
        read word
        if [ "${word}" == "y" ];
        then
                systemctl stop firewalld
                echo "firewalld를 다운하였습니다."
        else
                echo "firewalld가 기동된 상태로 시작합니다."
        fi
else
        echo "firewalld가 다운되어 있습니다."
fi

echo "#####"
echo "##### Check the Selinux"
echo "#####"
getSelinux=$(getenforce)
if [ "$getSelinux" = "Disabled" ];
then
        echo "Selinux가 다운되어 있습니다."
elif [ "$getSelinux" = "Permissive" ];
then
        echo "Selinux가 Permissive 모드로 기동되어 있습니다."
elif [ "$getSelinux" = "Enforcing" ];
then
        echo "Selinux가 기동되어 있습니다."
        echo "Selinux를 Permissive 모드로 변경 하시겠습니까? (y/n)"
        read fword
        if [ "${fword}" = "y" ];
        then
                setenfoce 0;
                echo "Selinux를 Permissive 모드로 변경 하였습니다."
        else
                echo "Selinux를 Enforce 모드로 유지하고 기동합니다."
        fi
fi

echo "#####"
echo "##### Check the SwapMemory"
echo "#####"
swapmem=$(free -m | grep Swap | awk '{print $2}')
if [ "$swapmem" = "0" ];
then
        echo "Swap Memory 영역이 해제되어 있습니다."
else
        echo "Kubernetes를 기동하기 위해 Swap Memory 영역을 해제 합니다."
        swapoff -a
fi

echo "#####"
echo "##### Initialize Kubernetes Master Node"
echo "#####"
kubeadm init --pod-network-cidr=10.2.0.0/16 --apiserver-advertise-address=$MASTER_NODE --image-repository="$DOCKER_REGISTRY/k8s" --kubernetes-version=v1.15.1 | tee init.txt
egrep "kubeadm join|--discovery-token-ca-cert-hash" init.txt > join.txt
workerJoin=$(sed 's/\\//' join.txt)

echo "#####"
echo "##### Configuration Kubernetes Master Node Security ENV"
echo "#####"
sh 1.kubeenv.sh;
echo "security 설정을 위한 초기 작업을 수행합니다."

echo "#####"
echo "##### Connect Private Docker Registry"
echo "#####"
kubectl create secret docker-registry regcred --docker-server=http://$DOCKER_REGISTRY --docker-username=admin --docker-password=admin123

echo "#####"
echo "##### Configuration Kubernetes Master Node Network & Dashboard"
echo "#####"
sh 2.kubectlapply.sh;
docker pull $DOCKER_REGISTRY/k8s/kubernetesui/dashboard:v2.0.0-beta1
docker pull $DOCKER_REGISTRY/k8s/kubernetesui/metrics-scraper:v1.0.0
docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-kube:2.5.2
docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-npc:2.5.2

echo "#####"
echo "##### Create Kubernetes Master Node Security Key"
echo "#####"
sh 3.securitykey.sh > security.txt
grep "token:" security.txt > token.txt
mv token.txt ~/encrypt/
echo "security key를 생성하였습니다."

echo "#####"
echo "##### Input your Kubernetes Dashboard Password"
echo "#####"
echo "인증서 비밀번호를 입력하세요 : "
sh 4.encrypt.sh
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "******************************************** Kubernetes Master Node 설정을 완료하였습니다. **********************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
sleep 5
echo ""
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "********************************************* Kubernetes Worker Node 구성을 시작합니다. *************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "#####"
echo "##### Check the SwapMemory"
echo "#####"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "swapoff -a"
echo "swap 영역을 해제 하였습니다."

echo "#####"
echo "##### Confirm Master Join info"
echo "#####"
echo $workerJoin

echo "#####"
echo "##### Join Kubernetes Master Node"
echo "#####"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE $workerJoin
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/pause:3.1"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/kube-proxy:v1.15.1"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-kube:2.5.2"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-npc:2.5.2"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0"
#sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/apache/my-apache2:latest"

echo "#####"
echo "##### Configuration Kubernetes Deployment, Pod, ReplicaSet"
echo "#####"
kubectl apply -f 5.new-apache-docker.yaml;
echo "#####"
echo "##### Configuration Kubernetes Service"
echo "#####"
kubectl apply -f 6.new-internal-service.yaml
echo "#####"
echo "##### Configuration Kubernetes Ingress Controller"
echo "#####"
kubectl apply -f 7.new-mandatory.yaml
echo "#####"
echo "##### Configuration Kubernetes Ingress, Role & Domain"
echo "#####"
echo "외부에서 접속할 도메인을 입력하세요 : "
read DOMAIN_URL
sed "s/test.nrson.go.kr/$DOMAIN_URL/" 8.new-ingress.yaml > 8.new-ingress2.yaml
kubectl apply -f 8.new-ingress2.yaml
echo "#####"
echo "##### Configuration Kubernetes External IP"
echo "#####"
kubectl apply -f 9-1.new-ingress-svc-externalip.yaml
echo "#####"
echo "##### Delete temporary files"
echo "#####"
rm -rf init.txt join.txt net2.yaml recommended2.yaml security.txt 8.new-ingress2.yaml
echo ""
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "******************************************* Kubernetes Worker Node 구성을 완료하였습니다. ***********************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo ""
echo "README"
echo "========================================================================================================================================="
echo "== 모든 구성이 완료되었습니다. =========================================================================================================="
echo "== 구성을 종료합니다. ==================================================================================================================="
echo "== ~/encrypt/ 하위의 ca.crt, kubecfg.p12, token.txt 파일을 대시보드에 접속하기 원하는 로컬에 다운로드 받으세요. ========================="
echo "== 개인키 등록 : C:\Windows\System32>certutil.exe -addstore "Root" [파일 경로]\ca.crt ==================================================="
echo "== 루트 인증 기관 등록 : certutil.exe -p dhfhskalsC! -user -importPFX [파일 경로]\kubecfg.p12 ==========================================="
echo "== Kubernetes Dashboard에 접속하여 구성 정보를 확인하시고, =============================================================================="
echo "== DashboardUrl (https://[master_ip]:6443/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login) =="
echo "== 테스트 페이지에 접속해 보세요. ======================================================================================================="
echo "== TestUrl (http://[DOMAIN_URL]/index.html) / Worker Node IP를 Hosts File에 등록하세요. ========================================"
echo "========================================================================================================================================="
EndTime=$(date +%s)
echo "Kubernetes 기동 및 설정 시간이 총 $(($EndTime - $StartTime))seconds 소요되었습니다."

- Kubernetes Master Node Setting

a) Set Env

b) Check the Firewall

c) Check the Selinux

d) Check the SwapMemory

e) Initialize Kubernetes Master Node

f) Configuration Kubernetes Master Node Security ENV

g) Connect Private Docker Registry

h) Configuration Kubernetes Master Node Network & Dashboard

i) Create Kubernetes Master Node Security Key

j) Input your Kubernetes Dashboard Password

- Kubernetes Worker Node Setting

k) Check the SwapMemory

l) Confirm Master Join info

m) Join Kubernetes Master Node

n) Configuration Kubernetes Deployment, Pod, ReplicaSet

o) Configuration Kubernetes Service

p) Configuration Kubernetes Ingress Controller

q) Configuration Kubernetes Ingress, Role & Domain

r) Configuration Kubernetes External IP

s) Delete temporary files

 

총 Master Node 10Step + Worker Node 9Step으로 이루어져 있습니다.

하나씩 살펴보고 적용해 실제로 적용해 보겠습니다.

3. Master Node 구성

a) Set Env

echo "#####" 
echo "##### Set ENV" 
echo "#####" 
export MASTER_NODE=192.168.xxx.xxx 
export WORKER_NODE=192.168.xxx.xxx 
export DOCKER_REGISTRY="192.168.xxx.xxx:port" 

→ 유일하게 직접 수정해야 하는 부분입니다.

Kubernetes Dashboard가 기동되는 Master Node IP와 Pod가 기동되는 Worker Node IP를 설정합니다.

또한 폐쇄망일 경우 DOCKER_REGISTRY에 IP:PORT를 설정합니다.

 

b) Check the Firewall

echo "#####" 
echo "##### Check the firewall" 
echo "#####" 
count=$(systemctl status firewalld | grep dead | wc -l) 
if(( "${count}" == 0 )); 
then 
        echo "firewalld이 기동되어 있습니다." 
        echo "firewalld를 다운하시겠습니까? (y/n)" 
        read word 
        if [ "${word}" == "y" ]; 
        then 
                systemctl stop firewalld 
                echo "firewalld를 다운하였습니다." 
        else 
                echo "firewalld를 기동된 상태로 시작합니다." 
        fi 
else 
        echo "firewalld가 다운되어 있습니다." 
fi

→ firewalld에 대한 설정을 검증합니다. firewalld 설정을 확인하고 기동되어 있을 경우 다운여부를 결정합니다.

 

c) Check the Selinux

echo "#####"
echo "##### Check the Selinux"
echo "#####"
getSelinux=$(getenforce)
if [ "$getSelinux" = "Disabled" ];
then
        echo "Selinux가 다운되어 있습니다."
elif [ "$getSelinux" = "Permissive" ];
then
        echo "Selinux가 Permissive 모드로 기동되어 있습니다."
elif [ "$getSelinux" = "Enforcing" ];
then
        echo "Selinux가 기동되어 있습니다."
        echo "Selinux를 Permissive 모드로 변경 하시겠습니까? (y/n)"
        read fword
        if [ "${fword}" = "y" ];
        then
                setenfoce 0;
                echo "Selinux를 Permissive 모드로 변경 하였습니다."
        else
                echo "Selinux를 Enforce 모드로 유지하고 기동합니다."
        fi
fi

→ selinux에 대한 설정을 검증합니다. selinux 설정을 확인하고 기동되어 있을 경우 다운여부를 결정합니다.


d) Check the SwapMemory

echo "#####" 
echo "##### Check the SwapMemory" 
echo "#####" 
swapmem=$(free -m | grep Swap | awk '{print $2}') 
if [ "$swapmem" = "0" ]; 
then 
        echo "Swap Memory 영역이 해제되어 있습니다." 
else 
        echo "Kubernetes를 기동하기 위해 Swap Memory 영역을 해제 합니다." 
        swapoff -a 
fi 

→ Kubernetes는 기동할 때 Swap 영역의 메모리를 사용할 수 없습니다. 이에 기동 전 Swap Memory 활성화 여부를 검토하고 영역 해제를 진행합니다.

 

e) Initialize Kubernetes Master Node

echo "#####" 
echo "##### Initialize Kubernetes Master Node" 
echo "#####" 
kubeadm init --pod-network-cidr=10.2.0.0/16 --apiserver-advertise-address=$MASTER_NODE --image-repository="$DOCKER_REGISTRY/k8s" --kubernetes-version=v1.15.1 | tee init.txt 
egrep "kubeadm join|--discovery-token-ca-cert-hash" init.txt > join.txt 
workerJoin=$(sed 's/\\//' join.txt) 

→ Kubernetes Master Node를 초기화 합니다.

초기화 시점에 생성되는 WorkNode Join 정보를 workerJoin 변수에 저장한 후 이후 Work Node에서 Master Node에 접속할 때 정보로 사용합니다.

--image-repository의 경우 기본 dockerhub를 사용하며, 로컬 레포지토리를 구축하였을 경우 변수로 넣어 주도록 합니다.

 

f) Configuration Kubernetes Master Node Security ENV

echo "#####" 
echo "##### Configuration Kubernetes Master Node Security ENV" 
echo "#####" 
sh 1.kubeenv.sh; 
echo "security 설정을 위한 초기 작업을 수행합니다." 

→ 다음으로 1.kubeenv.sh을 실행하여 Security 정보를 갱신하기 위한 .kube 디렉토리를 구성합니다.

 

g) Connect Private Docker Registry

echo "#####" 
echo "##### Connect Private Docker Registry" 
echo "#####" 
kubectl create secret docker-registry regcred --docker-server=http://$DOCKER_REGISTRY --docker-username=admin --docker-password=admin123 

→ Docker Private Registry를 구축한 경우 secret key를 생성하여 이후 접근하는 Docker 이미지들의 경로를 지정해 줄 수 있도록 설정합니다.

물론 Dockerhub를 사용할 경우에는 --docker-server 정보를 제외하거나 전체를 제외할 수 있습니다.

 

h) Configuration Kubernetes Master Node Network & Dashboard

echo "#####" 
echo "##### Configuration Kubernetes Master Node Network & Dashboard" 
echo "#####" 
sh 2.kubectlapply.sh; 
docker pull $DOCKER_REGISTRY/k8s/kubernetesui/dashboard:v2.0.0-beta1 
docker pull $DOCKER_REGISTRY/k8s/kubernetesui/metrics-scraper:v1.0.0 
docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-kube:2.5.2 
docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-npc:2.5.2 

→ Kubernetes Dashboard와 Network를 기동합니다. Docker Registry가 별도로 구축되어 있지 않은 경우 하단의 내용은 제외할 수 있습니다.

 

i) Create Kubernetes Master Node Security Key

echo "#####" 
echo "##### Create Kubernetes Master Node Security Key" 
echo "#####" 
sh 3.securitykey.sh > security.txt 
grep "token:" security.txt > token.txt 
mv token.txt ~/encrypt/ 
echo "security key를 생성하였습니다." 

→ Kubernetes Dashboard에 접근하기 위한 인증서를 생성하고 ~/encrypt/ 디렉토리에 인증서와 비밀번호 파일을 복사합니다.

 

j) Input your Kubernetes Dashboard Password

echo "#####" 
echo "##### Input your Kubernetes Dashboard Password" 
echo "#####" 
echo "인증서 비밀번호를 입력하세요 : " 
sh 4.encrypt.sh 

→ Master Node 마지막 작업으로 인증서 비밀번호를 구성합니다.

 

4. Worker Node 구성

Worker Node는 sshpass 명령어를 통해 구성하는 방식입니다.

 

k) Check the SwapMemory

echo "#####" 
echo "##### Check the SwapMemory" 
echo "#####" 
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "swapoff -a" 
echo "swap 영역을 해제 하였습니다." 

→ 먼저 마찬가지로 Swap Memory 영역을 해제합니다.

 

l) Confirm Master Join info

echo "#####" 
echo "##### Confirm Master Join info" 
echo "#####" 
echo $workerJoin 

→ 앞서 Master Node에서 생성한 workerJoin 정보를 확인합니다.


m) Join Kubernetes Master Node

echo "#####" 
echo "##### Join Kubernetes Master Node" 
echo "#####" 
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE $workerJoin 
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/pause:3.1" 
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/kube-proxy:v1.15.1" 
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-kube:2.5.2" 
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-npc:2.5.2" 
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0" 
#sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/apache/my-apache2:latest" 

→ Worker Node를 Master Node에 Join합니다.

Docker Repository를 별도로 구축하지 않은 경우 하단의 구성 정보는 필요하지 않습니다.

 

n) Configuration Kubernetes Deployment, Pod, ReplicaSet

echo "#####" 
echo "##### Configuration Kubernetes Deployment, Pod, ReplicaSet" 
echo "#####" 
kubectl apply -f 5.new-apache-docker.yaml; 

→ Kubernetes Deployment, Pod, ReplicaSet을 적용합니다.

 

o) Configuration Kubernetes Service

echo "#####" 
echo "##### Configuration Kubernetes Service" 
echo "#####" 
kubectl apply -f 6.new-internal-service.yaml 

→ Kubernetes Service를 추가합니다.

 

p) Configuration Kubernetes Ingress Controller

echo "#####" 
echo "##### Configuration Kubernetes Ingress Controller" 
echo "#####" 
kubectl apply -f 7.new-mandatory.yaml 

→ Ingress Controller를 추가합니다.

 

q) Configuration Kubernetes Ingress, Role & Domain

echo "#####" 
echo "##### Configuration Kubernetes Ingress, Role & Domain" 
echo "#####" 
echo "외부에서 접속할 도메인을 입력하세요 : " 
read DOMAIN_URL 
sed "s/test.nrson.go.kr/$DOMAIN_URL/" 8.new-ingress.yaml > 8.new-ingress2.yaml 
kubectl apply -f 8.new-ingress2.yaml 

→ Ingress, Role을 추가하며, 외부에 서비스할 도메인을 입력합니다.

 

r) Configuration Kubernetes External IP

echo "#####" 
echo "##### Configuration Kubernetes External IP" 
echo "#####" 
kubectl apply -f 9-1.new-ingress-svc-externalip.yaml 

→ 외부에 노출할 External IP를 설정합니다.

 

s) Delete temporary files

echo "#####" 
echo "##### Delete temporary files" 
echo "#####" 
rm -rf init.txt join.txt net2.yaml recommended2.yaml security.txt 8.new-ingress2.yaml 

→ 마지막으로 구성하며 생성된 파일을 정리하며 마무리합니다.

Kubernetes Boot & Setting

그럼 실제로 기동해 보도록 하겠습니다.

기동에 앞서 마찬가지로 본 포스팅을 위해 인터넷 환경에서 구성을 원하는 분들을 위한 0.kubernetesOneStepBootByInternet.sh을 추가로 생성하여 github에 commit 해 두도록 하겠습니다.

[root@kubemaster kubernetes_onestepboot]# sh 0.kubernetesOneStepBootByInternet.sh 
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
********************************************* Kubernetes Master Node 구성을 시작합니다. *************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
#####
##### Set ENV
#####
#####
##### Check the firewall
#####
firewalld가 기동되어 있습니다.
firewalld를 다운 하시겠습니까? (y/n)
y
firewalld를 다운하였습니다.
#####
##### Check the Selinux
#####
Selinux가 다운되어 있습니다.
#####
##### Check the SwapMemory
#####
Kubernetes를 기동하기 위해 Swap Memory 영역을 해제 합니다.
#####
##### Initialize Kubernetes Master Node
#####
[init] Using Kubernetes version: v1.15.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kubemaster localhost] and IPs [192.168.56.102 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kubemaster localhost] and IPs [192.168.56.102 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubemaster kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.56.102]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 37.007350 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node kubemaster as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node kubemaster as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: bos3m9.0ncjrroq0i6pnge7
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.56.102:6443 --token bos3m9.0ncjrroq0i6pnge7 \
    --discovery-token-ca-cert-hash sha256:8e8495ef0096544648fb5cc6d023f279e336c2a21d56e55be62af7167448989c 
#####
##### Configuration Kubernetes Master Node Security ENV
#####
security 설정을 위한 초기 작업을 수행합니다.
#####
##### Connect Private Docker Registry
#####
secret/regcred created
#####
##### Configuration Kubernetes Master Node Network & Dashboard
#####
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.extensions/weave-net created
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/kubernetes-metrics-scraper created
node/kubemaster untainted
#####
##### Create Kubernetes Master Node Security Key
#####
security key를 생성하였습니다.
#####
##### Input your Kubernetes Dashboard Password
#####
인증서 비밀번호를 입력하세요 : 
Enter Export Password:
Verifying - Enter Export Password:
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
******************************************** Kubernetes Master Node 설정을 완료하였습니다. **********************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************

*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
********************************************* Kubernetes Worker Node 구성을 시작합니다. *************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
#####
##### Check the SwapMemory
#####
swap 영역을 해제 하였습니다.
#####
##### Confirm Master Join info
#####
kubeadm join 192.168.56.102:6443 --token bos3m9.0ncjrroq0i6pnge7 --discovery-token-ca-cert-hash sha256:8e8495ef0096544648fb5cc6d023f279e336c2a21d56e55be62af7167448989c
#####
##### Join Kubernetes Master Node
#####
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Trying to pull repository docker.io/nara0617/my-apache2 ... 
latest: Pulling from docker.io/nara0617/my-apache2
Digest: sha256:f3b7575cb17091534b8916f49aa373c4e1f469338facc231e50c4b3a0aca0a9a
Status: Image is up to date for docker.io/nara0617/my-apache2:latest
#####
##### Configuration Kubernetes Deployment, Pod, ReplicaSet
#####
deployment.apps/apache-docker-deployment created
#####
##### Configuration Kubernetes Service
#####
service/apache-docker-service created
#####
##### Configuration Kubernetes Ingress Controller
#####
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created
#####
##### Configuration Kubernetes Ingress, Role & Domain
#####
외부에서 접속할 도메인을 입력하세요 : 
test.nrson.co.kr
ingress.extensions/nginx-ingress created
#####
##### Configuration Kubernetes External IP
#####
service/ingress-nginx created
#####
##### Delete temporary files
#####

*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
******************************************* Kubernetes Worker Node 구성을 완료하였습니다. ***********************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************

README
=========================================================================================================================================
== 모든 구성이 완료되었습니다. ==========================================================================================================
== 구성을 종료합니다. ===================================================================================================================
== ~/encrypt/ 하위의 ca.crt, kubecfg.p12, token.txt 파일을 대시보드에 접속하기 원하는 로컬에 다운로드 받으세요. =========================
== 개인키 등록 : C:\Windows\System32>certutil.exe -addstore Root [파일 경로]\ca.crt ===================================================
== 루트 인증 기관 등록 : certutil.exe -p dhfhskalsC! -user -importPFX [파일 경로]\kubecfg.p12 ===========================================
== Kubernetes Dashboard에 접속하여 구성 정보를 확인하시고, ==============================================================================
== DashboardUrl (https://[master_ip]:6443/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login) ==
== 테스트 페이지에 접속해 보세요. =======================================================================================================
== TestUrl (http://[DOMAIN_URL]/index.html) / Worker Node IP를 Hosts File에 등록하세요. ========================================
=========================================================================================================================================
Kubernetes 기동 및 설정 시간이 총 118seconds 소요되었습니다.
[root@kubemaster kubernetes_onestepboot]#

위와 같이 수행되며, 실행 과정에 몇가지 트러블 슈팅 포인트를 살펴보자면

a) docker ps -a 확인 시 etcd process가 재기동되는 현상

docker logs [ETCD Container ID] 확인 시 etcdmain: open /etc/kubernetes/pki/etcd/peer.crt: permission denied 에러가 발생할 다음 포스팅을 참고합니다.

[TroubleShooting] etcdmain: open /etc/kubernetes/pki/etcd/peer.crt: permission denied

 

b) sshpass를 통해 명령을 전송하였으나, 반응이 없을 경우

sshpass -p$WORKER_PASS ssh -o StrictHostKeyChecking=no root@$WORKER_NODE $workerJoin 와 같이 StrictHostKeyChecking=no 옵션을 추가합니다.

 

c) /var/log/messages 확인 시 err services "ingress-nginx" not found 발생

Aug 18 01:19:57 kubemaster journal: W0817 16:19:57.121256       8 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err services "ingress-nginx" not found 에러가 수없이 반복적으로 출력될 경우 다음 포스팅을 참고합니다.

[TroubleShooting] err services "ingress-nginx" not found

1. Kubernetes 0.kubernetesOneStepInfo.sh로 구동정보 확인하기

두번째로 0.kubernetesOneStepInfo.sh을 이용하여 기동이 완료되었는지 확인해 보도록 하겠습니다.

[root@kubemaster kubernetes_onestepboot]# cat 0.kubernetesOneStepInfo.sh 
echo "=========================================================================="
echo "==================Start Kubernetes Info==================================="
echo "=========================================================================="
echo "==================Kubernetes Deployment Info=============================="
kubectl get deployments --all-namespaces
echo "==================Kubernetes Pod Info====================================="
kubectl get pods --all-namespaces
echo "==================Kubernetes Replica Set Info============================="
kubectl get replicaset --all-namespaces
echo "==================Kubernetes Service Info================================="
kubectl get services --all-namespaces
echo "==================Kubernetes Ingress Info================================="
kubectl get ingress --all-namespaces
echo "==================Kubernetes Config Map Info=============================="
kubectl get configmap --all-namespaces
echo "=========================================================================="
echo "==================End Kubernetes Info====================================="
echo "=========================================================================="
[root@kubemaster kubernetes_onestepboot]#

Kubernetes가 정상 기동되었는지 여부를 확인하기 위한 명령어 모음 정도로 생각하면 됩니다.

[root@kubemaster kubernetes_onestepboot]# sh 0.kubernetesOneStepInfo.sh 
==========================================================================
==================Start Kubernetes Info===================================
==========================================================================
==================Kubernetes Deployment Info==============================
NAMESPACE              NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
default                apache-docker-deployment     2/2     2            2           18m
ingress-nginx          nginx-ingress-controller     1/1     1            1           18m
kube-system            coredns                      2/2     2            2           19m
kubernetes-dashboard   kubernetes-dashboard         1/1     1            1           19m
kubernetes-dashboard   kubernetes-metrics-scraper   1/1     1            1           19m
==================Kubernetes Pod Info=====================================
NAMESPACE              NAME                                          READY   STATUS    RESTARTS   AGE
default                apache-docker-deployment-6c44c89bfb-b9v87     1/1     Running   0          18m
default                apache-docker-deployment-6c44c89bfb-lqhwb     1/1     Running   0          18m
ingress-nginx          nginx-ingress-controller-6d88996bf-g26rh      1/1     Running   0          18m
kube-system            coredns-5c98db65d4-7qz7r                      1/1     Running   0          18m
kube-system            coredns-5c98db65d4-7ttdr                      1/1     Running   0          18m
kube-system            etcd-kubemaster                               1/1     Running   0          17m
kube-system            kube-apiserver-kubemaster                     1/1     Running   0          17m
kube-system            kube-controller-manager-kubemaster            1/1     Running   0          18m
kube-system            kube-proxy-rfd9b                              1/1     Running   0          18m
kube-system            kube-proxy-wxxhz                              1/1     Running   0          18m
kube-system            kube-scheduler-kubemaster                     1/1     Running   0          17m
kube-system            weave-net-hcmf7                               2/2     Running   0          18m
kube-system            weave-net-qrxgf                               2/2     Running   0          18m
kubernetes-dashboard   kubernetes-dashboard-67f4bff96d-fg9cl         1/1     Running   0          18m
kubernetes-dashboard   kubernetes-metrics-scraper-86456cdd8f-ptfnd   1/1     Running   0          18m
==================Kubernetes Replica Set Info=============================
NAMESPACE              NAME                                    DESIRED   CURRENT   READY   AGE
default                apache-docker-deployment-6c44c89bfb     2         2         2       18m
ingress-nginx          nginx-ingress-controller-6d88996bf      1         1         1       18m
kube-system            coredns-5c98db65d4                      2         2         2       18m
kubernetes-dashboard   kubernetes-dashboard-67f4bff96d         1         1         1       18m
kubernetes-dashboard   kubernetes-metrics-scraper-86456cdd8f   1         1         1       18m
==================Kubernetes Service Info=================================
NAMESPACE              NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP                     PORT(S)                  AGE
default                apache-docker-service       ClusterIP   10.103.14.107    <none>                          80/TCP                   18m
default                kubernetes                  ClusterIP   10.96.0.1        <none>                          443/TCP                  19m
ingress-nginx          ingress-nginx               ClusterIP   10.97.228.146    192.168.56.102,192.168.56.101   80/TCP                   17m
kube-system            kube-dns                    ClusterIP   10.96.0.10       <none>                          53/UDP,53/TCP,9153/TCP   19m
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP   10.103.160.52    <none>                          8000/TCP                 19m
kubernetes-dashboard   kubernetes-dashboard        ClusterIP   10.103.137.145   <none>                          443/TCP                  19m
==================Kubernetes Ingress Info=================================
NAMESPACE   NAME            HOSTS              ADDRESS                         PORTS   AGE
default     nginx-ingress   test.nrson.co.kr   192.168.56.101,192.168.56.102   80      17m
==================Kubernetes Config Map Info==============================
NAMESPACE              NAME                                 DATA   AGE
ingress-nginx          ingress-controller-leader-nginx      0      18m
ingress-nginx          nginx-configuration                  0      18m
ingress-nginx          tcp-services                         0      18m
ingress-nginx          udp-services                         0      18m
kube-public            cluster-info                         2      19m
kube-system            coredns                              1      19m
kube-system            extension-apiserver-authentication   6      19m
kube-system            kube-proxy                           2      19m
kube-system            kubeadm-config                       2      19m
kube-system            kubelet-config-1.15                  1      19m
kube-system            weave-net                            0      18m
kubernetes-dashboard   kubernetes-dashboard-settings        0      19m
==========================================================================
==================End Kubernetes Info=====================================
==========================================================================
[root@kubemaster kubernetes_onestepboot]# 

정상적으로 기동되었다면 위 수준의 정보가 출력되어야 합니다.

 

2. Kubernetes 0.kubernetesOneStepDown.sh로 Kubernetes 다운하기

마지막으로 0.kubernetesOneStepDown.sh을 이용하여 K8S를 내립니다.

[root@kubemaster kubernetes_onestepboot]# sh 0.kubernetesOneStepDown.sh 
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
W0818 01:42:04.663505   15489 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
[reset] Removing info for node "kubemaster" from the ConfigMap "kubeadm-config" in the "kube-system" Namespace
W0818 01:42:05.956989   24316 removeetcdmember.go:61] [reset] failed to remove etcd member: error syncing endpoints with etc: etcdclient: no available endpoints
.Please manually remove this etcd member using etcdctl
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
[root@kubemaster kubernetes_onestepboot]# sh 0.kubernetesOneStepInfo.sh 
==========================================================================
==================Start Kubernetes Info===================================
==========================================================================
==================Kubernetes Deployment Info==============================
The connection to the server 192.168.56.102:6443 was refused - did you specify the right host or port?
==================Kubernetes Pod Info=====================================
The connection to the server 192.168.56.102:6443 was refused - did you specify the right host or port?
==================Kubernetes Replica Set Info=============================
The connection to the server 192.168.56.102:6443 was refused - did you specify the right host or port?
==================Kubernetes Service Info=================================
The connection to the server 192.168.56.102:6443 was refused - did you specify the right host or port?
==================Kubernetes Ingress Info=================================
The connection to the server 192.168.56.102:6443 was refused - did you specify the right host or port?
==================Kubernetes Config Map Info==============================
The connection to the server 192.168.56.102:6443 was refused - did you specify the right host or port?
==========================================================================
==================End Kubernetes Info=====================================
==========================================================================
[root@kubemaster kubernetes_onestepboot]# 

쓰다보니 내용이 길어졌는데 요약하자면 다음과 같습니다.


1) Shell Script 실행을 위한 이동 (cd $GIT_REPO_HOME/kubernetesonestepboot/kubernetes_onestepboot/)

2) Shell Script 실행 (sh 0.kubernetesOneStepBootByInternet.sh)

3) Shell Script로 기동 상태 확인 (sh 0.kubernetesOneStepInfo.sh)

4) Shell Script로 프로세스 다운 (sh 0.kubernetesOneStepDown.sh)


정말 간단하죠?

그럼 총 정리를 해보도록 하겠습니다.

지난 포스팅에서 살펴보았던 Kubernetes 설치 과정은

1) Virtual Box 및 CentOS 설치

2) git 설치 (yum install -y git)

3) git repository 다운로드 (git clone https://github.com/sonnaraon/kubernetesonestepboot.git)

4) Shell Script 실행을 위한 이동 (cd $GIT_REPO_HOME/kubernetesonestepboot/kubernetes_onestepinstall/)

5) Shell Script 실행 (sh 1.kubernetesOneStepInstallByInternet.sh)

+

Kubernetes 설정/기동 및 확인 과정은

1) Shell Script 실행을 위한 이동 (cd $GIT_REPO_HOME/kubernetesonestepboot/kubernetes_onestepboot/)

2) Shell Script 실행 (sh 0.kubernetesOneStepBootByInternet.sh)

3) Shell Script로 기동 상태 확인 (sh 0.kubernetesOneStepInfo.sh)

4) Shell Script로 프로세스 다운 (sh 0.kubernetesOneStepDown.sh)

로서 손쉽게 Kubernetes Project를 구성하고 운영할 수 있습니다.

물론 제공된 Yaml 파일을 잘 편집하여 본인이 원하는 새로운 Docker 이미지 환경으로 사용하셔도 무방합니다.

Kubernetes Dashboard 및 Application 호출

마지막으로 지금까지 구성한 Kubernetes Dashboard 및 Application 정보를 확인하고 호출해 보도록 하겠습니다.

1) ExternalIP로 직접 호출한 경우

화면이 위와 같이 404 Not Found와 함께 nginx version이 출력됩니다.

현재 Ingress Nginx에 적용된 Ingress Rule에 domain은 test.nrson.co.kr로 되어 있기 때문에 ip로는 직접 접근이 불가능합니다.

2) Ingress에 등록한 Domain으로 호출한 경우

당연하겠지만, Domain 정보를 hosts 파일에 등록해야하며, 위와 같이 정상적으로 출력되는 것을 확인할 수 있습니다.

3) Kubernetes Dashboard

위와 같이 Workload를 통해 생성된 apache-docker-deployment pod 2개와 각각의 정보들을 왼쪽 메뉴바를 통해 확인할 수 있습니다.

마지막으로 ingress-nginx의 Namespace 정보입니다.

위와 같이 nginx-ingress-controller 정보와 각종 룰 정보들을 확인할 수 있습니다.

포스팅이 굉장히 길어지긴 했지만, 이와 같은 과정으로 Kubernetes를 손쉽게 구축해 볼 수 있도록 가이드를 작성해 보았습니다.

차근차근 곱씹어 보면서 yaml 파일이나 Shell Script도 직접 수정해 보고 이것 저것 테스트 해보면서 스킬업 해 나가시기 바랍니다.

728x90
반응형