티스토리 뷰
③ 클라우드/ⓚ Kubernetes
[Container Management] Kubernetes OneStep Install & Boot & Setting (Part 2. Boot & Setting)
GodNR 2019. 8. 17. 17:54728x90
반응형
지금부터 살펴볼 내용은 어쩌면 지금까지 살펴본 다양한 Kubernetes 포스팅의 완결판이 될 수도 있을 것 같습니다.
지금까지 다양한 Component를 살펴보며 Kubernetes가 제공하는 Request Flow에 대해 알아보고 구성해 보았습니다.
다만 각 컴포넌트들을 직접 구성하기에는 자세히 설명을 하였지만 처음 접하는 분들에게는 어려운 과정이 될 것입니다.
이에 누구나 손쉽게 따라 구성해 볼 수 있게 Shell Script와 YAML 파일을 작성해 보았습니다.
이전 포스팅은 아래를 참고해 주시기 바랍니다.
[Container Management] Kubernetes Master Node 설치
[Container Management] Kubernetes Dashboard Install & Setting
[Container Management] Kubernetes Woker Node Install & Setting
[Container Management] Kubernetes Pod 생성 가이드
[Container Management] Kubernetes Service 생성 가이드
[Container Management] Kubernetes 애플리케이션 관리
[Container Management] Kubernetes ingress & ingress controller
특히 지난 Part 1. Install에서 살펴본 OneStep Install 과정은 다음을 참고해 주세요.
[Container Management] Kubernetes OneStep Install & Boot & Setting (Part 1. Install)
Kubernetes One Step Boot & Setting
이번 포스팅에서는 지난 시간에 과제로 드린 Master Node & Worker Node가 동일하게 구성되어 있다는 가정하에 진행하도록 하겠습니다.
혹시나 내용을 스킵하고 빠르게 수행해 보고 싶으신 분은 제일 하단으로 이동하시기 바랍니다.
1. Git Repository Clone
먼저 진행 할 작업은 사전에 만들어 둔 github repository에 있는 소스를 다운로드 받습니다.
앞선 포스팅에서 이미 git clone을 받으신 분은 넘어가셔도 됩니다.
- git install
[root@kubemaster ~]# yum install -y git
Last login: Sat Aug 17 17:13:52 2019
Loaded plugins: fastestmirror
Determining fastest mirrors
* base: data.aonenetworks.kr
* extras: data.aonenetworks.kr
* updates: data.aonenetworks.kr
base | 3.6 kB 00:00:00
extras | 3.4 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/4): base/7/x86_64/primary_db | 6.0 MB 00:00:00
(2/4): base/7/x86_64/group_gz | 166 kB 00:00:03
(3/4): extras/7/x86_64/primary_db | 215 kB 00:00:04
...
...
...
Installed:
git.x86_64 0:1.8.3.1-20.el7
Dependency Installed:
perl.x86_64 4:5.16.3-294.el7_6 perl-Carp.noarch 0:1.26-244.el7 perl-Encode.x86_64 0:2.51-7.el7 perl-Error.noarch 1:0.17020-2.el7 perl-Exporter.noarch 0:5.68-3.el7
perl-File-Path.noarch 0:2.09-2.el7 perl-File-Temp.noarch 0:0.23.01-3.el7 perl-Filter.x86_64 0:1.49-3.el7 perl-Getopt-Long.noarch 0:2.40-3.el7 perl-Git.noarch 0:1.8.3.1-20.el7
perl-HTTP-Tiny.noarch 0:0.033-3.el7 perl-PathTools.x86_64 0:3.40-5.el7 perl-Pod-Escapes.noarch 1:1.04-294.el7_6 perl-Pod-Perldoc.noarch 0:3.20-4.el7 perl-Pod-Simple.noarch 1:3.28-4.el7
perl-Pod-Usage.noarch 0:1.63-3.el7 perl-Scalar-List-Utils.x86_64 0:1.27-248.el7 perl-Socket.x86_64 0:2.010-4.el7 perl-Storable.x86_64 0:2.45-3.el7 perl-TermReadKey.x86_64 0:2.30-20.el7
perl-Text-ParseWords.noarch 0:3.29-4.el7 perl-Time-HiRes.x86_64 4:1.9725-3.el7 perl-Time-Local.noarch 0:1.2300-2.el7 perl-constant.noarch 0:1.27-2.el7 perl-libs.x86_64 4:5.16.3-294.el7_6
perl-macros.x86_64 4:5.16.3-294.el7_6 perl-parent.noarch 1:0.225-244.el7 perl-podlators.noarch 0:2.5.1-3.el7 perl-threads.x86_64 0:1.87-4.el7 perl-threads-shared.x86_64 0:1.43-6.el7
rsync.x86_64 0:3.1.2-6.el7_6.1
Complete!
[root@kubemaster ~]#- git clone
[root@kubemaster ~]# mkdir git_repo
[root@kubemaster ~]# cd git_repo/
[root@kubemaster git_repo]# ls
[root@kubemaster git_repo]# git clone https://github.com/sonnaraon/kubernetesonestepboot.git
Cloning into 'kubernetesonestepboot'...
remote: Enumerating objects: 149, done.
remote: Counting objects: 100% (149/149), done.
remote: Compressing objects: 100% (89/89), done.
remote: Total 149 (delta 57), reused 146 (delta 54), pack-reused 0
Receiving objects: 100% (149/149), 32.82 KiB | 0 bytes/s, done.
Resolving deltas: 100% (57/57), done.
[root@kubemaster git_repo]# ls -la
drwxr-xr-x. 3 root root 35 817 11:18 .
dr-xr-x---. 4 root root 142 817 11:18 ..
drwxr-xr-x. 5 root root 100 817 11:18 kubernetesonestepboot
[root@kubemaster git_repo]# cd kubernetesonestepboot/
[root@kubemaster kubernetesonestepboot]# ls
ReadMe.adoc kubernetes_onestepboot kubernetes_onestepinstall
[root@kubemaster kubernetesonestepboot]# ls -la
drwxr-xr-x. 5 root root 100 817 11:18 .
drwxr-xr-x. 3 root root 35 817 11:18 ..
drwxr-xr-x. 8 root root 163 817 11:18 .git
-rw-r--r--. 1 root root 4212 817 11:18 ReadMe.adoc
drwxr-xr-x. 2 root root 4096 817 11:18 kubernetes_onestepboot
drwxr-xr-x. 2 root root 95 817 11:18 kubernetes_onestepinstall
[root@kubemaster kubernetesonestepboot]#아래서 설명해 드리겠지만, 다운로드 받은 git repository kubernetesonestepboot project는
1) kubernetes_onestepinstall : kubernetes 설치용 Shell Script
2) kubernetes_onestepboot : kubernetes 기동 및 기본 환경 구성용 Shell Script
로 구성되어 있습니다.
kubernetes_onestepboot 디렉토리 하위로 이동합니다.
[root@kubemaster kubernetes_onestepboot]# ls -la
total 104
drwxr-xr-x. 2 root root 4096 Aug 17 17:20 .
drwxr-xr-x. 5 root root 100 Aug 17 17:20 ..
-rwxr-xr-x. 1 root root 11175 Aug 17 17:20 0.kubernetesOneStepBoot.sh
-rwxr-xr-x. 1 root root 69 Aug 17 17:20 0.kubernetesOneStepDown.sh
-rwxr-xr-x. 1 root root 1213 Aug 17 17:20 0.kubernetesOneStepInfo.sh
-rw-r--r--. 1 root root 142 Aug 17 17:20 1.kubeenv.sh
-rw-r--r--. 1 root root 419 Aug 17 17:20 2.kubectlapply.sh
-rw-r--r--. 1 root root 540 Aug 17 17:20 3.securitykey.sh
-rw-r--r--. 1 root root 413 Aug 17 17:20 4.encrypt.sh
-rw-r--r--. 1 root root 748 Aug 17 17:20 5.new-apache-docker.yaml
-rw-r--r--. 1 root root 674 Aug 17 17:20 5.new-apache-docker.yaml_org
-rw-r--r--. 1 root root 217 Aug 17 17:20 6.new-internal-service.yaml
-rw-r--r--. 1 root root 6067 Aug 17 17:20 7.new-mandatory.yaml
-rw-r--r--. 1 root root 6045 Aug 17 17:20 7.new-mandatory.yaml_org
-rw-r--r--. 1 root root 357 Aug 17 17:20 8.new-ingress.yaml
-rw-r--r--. 1 root root 339 Aug 17 17:20 9-1.new-ingress-svc-externalip.yaml
-rw-r--r--. 1 root root 306 Aug 17 17:20 9-2.ingress-svc-nodeport.yaml
-rw-r--r--. 1 root root 1143 Aug 17 17:20 9-3.default-backend.yaml
-rw-r--r--. 1 root root 3230 Aug 17 17:20 kubernetes_masternode_setting.txt
-rw-r--r--. 1 root root 6784 Aug 17 17:20 net.yaml
-rw-r--r--. 1 root root 6928 Aug 17 17:20 recommended.yaml
[root@kubemaster kubernetes_onestepboot]#install 시에는 0.kubernetesOneStepInstall.sh (or 1.kubernetesOneStepInstallByInternet.sh) 하나의 실행 파일이 존재하였지만, boot Script에는 총 3개의 실행 파일이 존재합니다.
- 0.kubernetesOneStepBoot.sh : Kubernetes Master Node, Worker Node 기동, Component 생성 및 샘플 애플리케이션 생성 등
- 0.kubernetesOneStepDown.sh : Kubernetes Master Node, Worker Node 다운
- 0.kubernetesOneStepInfo.sh : Kubernetes 기동 이후 각종 정보 확인
2. Kubernetes 0.kubernetesOneStepBoot.sh로 구동 및 설정하기
먼저 살펴볼 정보는 0.kubernetesOneStepBoot.sh입니다.
#!/bin/bash
StartTime=$(date +%s)
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "********************************************* Kubernetes Master Node 구성을 시작합니다. *************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "#####"
echo "##### Set ENV"
echo "#####"
export MASTER_NODE=192.168.xxx.xxx
export WORKER_NODE=192.168.xxx.xxx
export WORKER_PASS=xxxxxx
export DOCKER_REGISTRY="192.168.xxx.xxx:port"
echo "#####"
echo "##### Check the firewall"
echo "#####"
count=$(systemctl status firewalld | grep dead | wc -l)
if(( "${count}" == 0 ));
then
echo "firewalld가 기동되어 있습니다."
echo "firewalld를 다운 하시겠습니까? (y/n)"
read word
if [ "${word}" == "y" ];
then
systemctl stop firewalld
echo "firewalld를 다운하였습니다."
else
echo "firewalld가 기동된 상태로 시작합니다."
fi
else
echo "firewalld가 다운되어 있습니다."
fi
echo "#####"
echo "##### Check the Selinux"
echo "#####"
getSelinux=$(getenforce)
if [ "$getSelinux" = "Disabled" ];
then
echo "Selinux가 다운되어 있습니다."
elif [ "$getSelinux" = "Permissive" ];
then
echo "Selinux가 Permissive 모드로 기동되어 있습니다."
elif [ "$getSelinux" = "Enforcing" ];
then
echo "Selinux가 기동되어 있습니다."
echo "Selinux를 Permissive 모드로 변경 하시겠습니까? (y/n)"
read fword
if [ "${fword}" = "y" ];
then
setenfoce 0;
echo "Selinux를 Permissive 모드로 변경 하였습니다."
else
echo "Selinux를 Enforce 모드로 유지하고 기동합니다."
fi
fi
echo "#####"
echo "##### Check the SwapMemory"
echo "#####"
swapmem=$(free -m | grep Swap | awk '{print $2}')
if [ "$swapmem" = "0" ];
then
echo "Swap Memory 영역이 해제되어 있습니다."
else
echo "Kubernetes를 기동하기 위해 Swap Memory 영역을 해제 합니다."
swapoff -a
fi
echo "#####"
echo "##### Initialize Kubernetes Master Node"
echo "#####"
kubeadm init --pod-network-cidr=10.2.0.0/16 --apiserver-advertise-address=$MASTER_NODE --image-repository="$DOCKER_REGISTRY/k8s" --kubernetes-version=v1.15.1 | tee init.txt
egrep "kubeadm join|--discovery-token-ca-cert-hash" init.txt > join.txt
workerJoin=$(sed 's/\\//' join.txt)
echo "#####"
echo "##### Configuration Kubernetes Master Node Security ENV"
echo "#####"
sh 1.kubeenv.sh;
echo "security 설정을 위한 초기 작업을 수행합니다."
echo "#####"
echo "##### Connect Private Docker Registry"
echo "#####"
kubectl create secret docker-registry regcred --docker-server=http://$DOCKER_REGISTRY --docker-username=admin --docker-password=admin123
echo "#####"
echo "##### Configuration Kubernetes Master Node Network & Dashboard"
echo "#####"
sh 2.kubectlapply.sh;
docker pull $DOCKER_REGISTRY/k8s/kubernetesui/dashboard:v2.0.0-beta1
docker pull $DOCKER_REGISTRY/k8s/kubernetesui/metrics-scraper:v1.0.0
docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-kube:2.5.2
docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-npc:2.5.2
echo "#####"
echo "##### Create Kubernetes Master Node Security Key"
echo "#####"
sh 3.securitykey.sh > security.txt
grep "token:" security.txt > token.txt
mv token.txt ~/encrypt/
echo "security key를 생성하였습니다."
echo "#####"
echo "##### Input your Kubernetes Dashboard Password"
echo "#####"
echo "인증서 비밀번호를 입력하세요 : "
sh 4.encrypt.sh
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "******************************************** Kubernetes Master Node 설정을 완료하였습니다. **********************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
sleep 5
echo ""
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "********************************************* Kubernetes Worker Node 구성을 시작합니다. *************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "#####"
echo "##### Check the SwapMemory"
echo "#####"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "swapoff -a"
echo "swap 영역을 해제 하였습니다."
echo "#####"
echo "##### Confirm Master Join info"
echo "#####"
echo $workerJoin
echo "#####"
echo "##### Join Kubernetes Master Node"
echo "#####"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE $workerJoin
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/pause:3.1"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/kube-proxy:v1.15.1"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-kube:2.5.2"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-npc:2.5.2"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0"
#sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/apache/my-apache2:latest"
echo "#####"
echo "##### Configuration Kubernetes Deployment, Pod, ReplicaSet"
echo "#####"
kubectl apply -f 5.new-apache-docker.yaml;
echo "#####"
echo "##### Configuration Kubernetes Service"
echo "#####"
kubectl apply -f 6.new-internal-service.yaml
echo "#####"
echo "##### Configuration Kubernetes Ingress Controller"
echo "#####"
kubectl apply -f 7.new-mandatory.yaml
echo "#####"
echo "##### Configuration Kubernetes Ingress, Role & Domain"
echo "#####"
echo "외부에서 접속할 도메인을 입력하세요 : "
read DOMAIN_URL
sed "s/test.nrson.go.kr/$DOMAIN_URL/" 8.new-ingress.yaml > 8.new-ingress2.yaml
kubectl apply -f 8.new-ingress2.yaml
echo "#####"
echo "##### Configuration Kubernetes External IP"
echo "#####"
kubectl apply -f 9-1.new-ingress-svc-externalip.yaml
echo "#####"
echo "##### Delete temporary files"
echo "#####"
rm -rf init.txt join.txt net2.yaml recommended2.yaml security.txt 8.new-ingress2.yaml
echo ""
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "******************************************* Kubernetes Worker Node 구성을 완료하였습니다. ***********************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo "*****************************************************************************************************************************************"
echo ""
echo "README"
echo "========================================================================================================================================="
echo "== 모든 구성이 완료되었습니다. =========================================================================================================="
echo "== 구성을 종료합니다. ==================================================================================================================="
echo "== ~/encrypt/ 하위의 ca.crt, kubecfg.p12, token.txt 파일을 대시보드에 접속하기 원하는 로컬에 다운로드 받으세요. ========================="
echo "== 개인키 등록 : C:\Windows\System32>certutil.exe -addstore "Root" [파일 경로]\ca.crt ==================================================="
echo "== 루트 인증 기관 등록 : certutil.exe -p dhfhskalsC! -user -importPFX [파일 경로]\kubecfg.p12 ==========================================="
echo "== Kubernetes Dashboard에 접속하여 구성 정보를 확인하시고, =============================================================================="
echo "== DashboardUrl (https://[master_ip]:6443/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login) =="
echo "== 테스트 페이지에 접속해 보세요. ======================================================================================================="
echo "== TestUrl (http://[DOMAIN_URL]/index.html) / Worker Node IP를 Hosts File에 등록하세요. ========================================"
echo "========================================================================================================================================="
EndTime=$(date +%s)
echo "Kubernetes 기동 및 설정 시간이 총 $(($EndTime - $StartTime))seconds 소요되었습니다."- Kubernetes Master Node Setting
a) Set Env
b) Check the Firewall
c) Check the Selinux
d) Check the SwapMemory
e) Initialize Kubernetes Master Node
f) Configuration Kubernetes Master Node Security ENV
g) Connect Private Docker Registry
h) Configuration Kubernetes Master Node Network & Dashboard
i) Create Kubernetes Master Node Security Key
j) Input your Kubernetes Dashboard Password
- Kubernetes Worker Node Setting
k) Check the SwapMemory
l) Confirm Master Join info
m) Join Kubernetes Master Node
n) Configuration Kubernetes Deployment, Pod, ReplicaSet
o) Configuration Kubernetes Service
p) Configuration Kubernetes Ingress Controller
q) Configuration Kubernetes Ingress, Role & Domain
r) Configuration Kubernetes External IP
s) Delete temporary files
총 Master Node 10Step + Worker Node 9Step으로 이루어져 있습니다.
하나씩 살펴보고 적용해 실제로 적용해 보겠습니다.
3. Master Node 구성
a) Set Env
echo "#####"
echo "##### Set ENV"
echo "#####"
export MASTER_NODE=192.168.xxx.xxx
export WORKER_NODE=192.168.xxx.xxx
export DOCKER_REGISTRY="192.168.xxx.xxx:port" → 유일하게 직접 수정해야 하는 부분입니다.
Kubernetes Dashboard가 기동되는 Master Node IP와 Pod가 기동되는 Worker Node IP를 설정합니다.
또한 폐쇄망일 경우 DOCKER_REGISTRY에 IP:PORT를 설정합니다.
b) Check the Firewall
echo "#####"
echo "##### Check the firewall"
echo "#####"
count=$(systemctl status firewalld | grep dead | wc -l)
if(( "${count}" == 0 ));
then
echo "firewalld이 기동되어 있습니다."
echo "firewalld를 다운하시겠습니까? (y/n)"
read word
if [ "${word}" == "y" ];
then
systemctl stop firewalld
echo "firewalld를 다운하였습니다."
else
echo "firewalld를 기동된 상태로 시작합니다."
fi
else
echo "firewalld가 다운되어 있습니다."
fi→ firewalld에 대한 설정을 검증합니다. firewalld 설정을 확인하고 기동되어 있을 경우 다운여부를 결정합니다.
c) Check the Selinux
echo "#####"
echo "##### Check the Selinux"
echo "#####"
getSelinux=$(getenforce)
if [ "$getSelinux" = "Disabled" ];
then
echo "Selinux가 다운되어 있습니다."
elif [ "$getSelinux" = "Permissive" ];
then
echo "Selinux가 Permissive 모드로 기동되어 있습니다."
elif [ "$getSelinux" = "Enforcing" ];
then
echo "Selinux가 기동되어 있습니다."
echo "Selinux를 Permissive 모드로 변경 하시겠습니까? (y/n)"
read fword
if [ "${fword}" = "y" ];
then
setenfoce 0;
echo "Selinux를 Permissive 모드로 변경 하였습니다."
else
echo "Selinux를 Enforce 모드로 유지하고 기동합니다."
fi
fi→ selinux에 대한 설정을 검증합니다. selinux 설정을 확인하고 기동되어 있을 경우 다운여부를 결정합니다.
d) Check the SwapMemory
echo "#####"
echo "##### Check the SwapMemory"
echo "#####"
swapmem=$(free -m | grep Swap | awk '{print $2}')
if [ "$swapmem" = "0" ];
then
echo "Swap Memory 영역이 해제되어 있습니다."
else
echo "Kubernetes를 기동하기 위해 Swap Memory 영역을 해제 합니다."
swapoff -a
fi → Kubernetes는 기동할 때 Swap 영역의 메모리를 사용할 수 없습니다. 이에 기동 전 Swap Memory 활성화 여부를 검토하고 영역 해제를 진행합니다.
e) Initialize Kubernetes Master Node
echo "#####"
echo "##### Initialize Kubernetes Master Node"
echo "#####"
kubeadm init --pod-network-cidr=10.2.0.0/16 --apiserver-advertise-address=$MASTER_NODE --image-repository="$DOCKER_REGISTRY/k8s" --kubernetes-version=v1.15.1 | tee init.txt
egrep "kubeadm join|--discovery-token-ca-cert-hash" init.txt > join.txt
workerJoin=$(sed 's/\\//' join.txt) → Kubernetes Master Node를 초기화 합니다.
초기화 시점에 생성되는 WorkNode Join 정보를 workerJoin 변수에 저장한 후 이후 Work Node에서 Master Node에 접속할 때 정보로 사용합니다.
--image-repository의 경우 기본 dockerhub를 사용하며, 로컬 레포지토리를 구축하였을 경우 변수로 넣어 주도록 합니다.
f) Configuration Kubernetes Master Node Security ENV
echo "#####"
echo "##### Configuration Kubernetes Master Node Security ENV"
echo "#####"
sh 1.kubeenv.sh;
echo "security 설정을 위한 초기 작업을 수행합니다." → 다음으로 1.kubeenv.sh을 실행하여 Security 정보를 갱신하기 위한 .kube 디렉토리를 구성합니다.
g) Connect Private Docker Registry
echo "#####"
echo "##### Connect Private Docker Registry"
echo "#####"
kubectl create secret docker-registry regcred --docker-server=http://$DOCKER_REGISTRY --docker-username=admin --docker-password=admin123 → Docker Private Registry를 구축한 경우 secret key를 생성하여 이후 접근하는 Docker 이미지들의 경로를 지정해 줄 수 있도록 설정합니다.
물론 Dockerhub를 사용할 경우에는 --docker-server 정보를 제외하거나 전체를 제외할 수 있습니다.
h) Configuration Kubernetes Master Node Network & Dashboard
echo "#####"
echo "##### Configuration Kubernetes Master Node Network & Dashboard"
echo "#####"
sh 2.kubectlapply.sh;
docker pull $DOCKER_REGISTRY/k8s/kubernetesui/dashboard:v2.0.0-beta1
docker pull $DOCKER_REGISTRY/k8s/kubernetesui/metrics-scraper:v1.0.0
docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-kube:2.5.2
docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-npc:2.5.2 → Kubernetes Dashboard와 Network를 기동합니다. Docker Registry가 별도로 구축되어 있지 않은 경우 하단의 내용은 제외할 수 있습니다.
i) Create Kubernetes Master Node Security Key
echo "#####"
echo "##### Create Kubernetes Master Node Security Key"
echo "#####"
sh 3.securitykey.sh > security.txt
grep "token:" security.txt > token.txt
mv token.txt ~/encrypt/
echo "security key를 생성하였습니다." → Kubernetes Dashboard에 접근하기 위한 인증서를 생성하고 ~/encrypt/ 디렉토리에 인증서와 비밀번호 파일을 복사합니다.
j) Input your Kubernetes Dashboard Password
echo "#####"
echo "##### Input your Kubernetes Dashboard Password"
echo "#####"
echo "인증서 비밀번호를 입력하세요 : "
sh 4.encrypt.sh → Master Node 마지막 작업으로 인증서 비밀번호를 구성합니다.
4. Worker Node 구성
Worker Node는 sshpass 명령어를 통해 구성하는 방식입니다.
k) Check the SwapMemory
echo "#####"
echo "##### Check the SwapMemory"
echo "#####"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "swapoff -a"
echo "swap 영역을 해제 하였습니다." → 먼저 마찬가지로 Swap Memory 영역을 해제합니다.
l) Confirm Master Join info
echo "#####"
echo "##### Confirm Master Join info"
echo "#####"
echo $workerJoin → 앞서 Master Node에서 생성한 workerJoin 정보를 확인합니다.
m) Join Kubernetes Master Node
echo "#####"
echo "##### Join Kubernetes Master Node"
echo "#####"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE $workerJoin
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/pause:3.1"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/kube-proxy:v1.15.1"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-kube:2.5.2"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/weaveworks/weave-npc:2.5.2"
sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0"
#sshpass -p$WORKER_PASS ssh root@$WORKER_NODE "docker pull $DOCKER_REGISTRY/k8s/apache/my-apache2:latest" → Worker Node를 Master Node에 Join합니다.
Docker Repository를 별도로 구축하지 않은 경우 하단의 구성 정보는 필요하지 않습니다.
n) Configuration Kubernetes Deployment, Pod, ReplicaSet
echo "#####"
echo "##### Configuration Kubernetes Deployment, Pod, ReplicaSet"
echo "#####"
kubectl apply -f 5.new-apache-docker.yaml; → Kubernetes Deployment, Pod, ReplicaSet을 적용합니다.
o) Configuration Kubernetes Service
echo "#####"
echo "##### Configuration Kubernetes Service"
echo "#####"
kubectl apply -f 6.new-internal-service.yaml → Kubernetes Service를 추가합니다.
p) Configuration Kubernetes Ingress Controller
echo "#####"
echo "##### Configuration Kubernetes Ingress Controller"
echo "#####"
kubectl apply -f 7.new-mandatory.yaml → Ingress Controller를 추가합니다.
q) Configuration Kubernetes Ingress, Role & Domain
echo "#####"
echo "##### Configuration Kubernetes Ingress, Role & Domain"
echo "#####"
echo "외부에서 접속할 도메인을 입력하세요 : "
read DOMAIN_URL
sed "s/test.nrson.go.kr/$DOMAIN_URL/" 8.new-ingress.yaml > 8.new-ingress2.yaml
kubectl apply -f 8.new-ingress2.yaml → Ingress, Role을 추가하며, 외부에 서비스할 도메인을 입력합니다.
r) Configuration Kubernetes External IP
echo "#####"
echo "##### Configuration Kubernetes External IP"
echo "#####"
kubectl apply -f 9-1.new-ingress-svc-externalip.yaml → 외부에 노출할 External IP를 설정합니다.
s) Delete temporary files
echo "#####"
echo "##### Delete temporary files"
echo "#####"
rm -rf init.txt join.txt net2.yaml recommended2.yaml security.txt 8.new-ingress2.yaml → 마지막으로 구성하며 생성된 파일을 정리하며 마무리합니다.
Kubernetes Boot & Setting
그럼 실제로 기동해 보도록 하겠습니다.
기동에 앞서 마찬가지로 본 포스팅을 위해 인터넷 환경에서 구성을 원하는 분들을 위한 0.kubernetesOneStepBootByInternet.sh을 추가로 생성하여 github에 commit 해 두도록 하겠습니다.
[root@kubemaster kubernetes_onestepboot]# sh 0.kubernetesOneStepBootByInternet.sh
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
********************************************* Kubernetes Master Node 구성을 시작합니다. *************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
#####
##### Set ENV
#####
#####
##### Check the firewall
#####
firewalld가 기동되어 있습니다.
firewalld를 다운 하시겠습니까? (y/n)
y
firewalld를 다운하였습니다.
#####
##### Check the Selinux
#####
Selinux가 다운되어 있습니다.
#####
##### Check the SwapMemory
#####
Kubernetes를 기동하기 위해 Swap Memory 영역을 해제 합니다.
#####
##### Initialize Kubernetes Master Node
#####
[init] Using Kubernetes version: v1.15.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kubemaster localhost] and IPs [192.168.56.102 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kubemaster localhost] and IPs [192.168.56.102 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubemaster kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.56.102]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 37.007350 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node kubemaster as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node kubemaster as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: bos3m9.0ncjrroq0i6pnge7
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.56.102:6443 --token bos3m9.0ncjrroq0i6pnge7 \
--discovery-token-ca-cert-hash sha256:8e8495ef0096544648fb5cc6d023f279e336c2a21d56e55be62af7167448989c
#####
##### Configuration Kubernetes Master Node Security ENV
#####
security 설정을 위한 초기 작업을 수행합니다.
#####
##### Connect Private Docker Registry
#####
secret/regcred created
#####
##### Configuration Kubernetes Master Node Network & Dashboard
#####
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.extensions/weave-net created
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/kubernetes-metrics-scraper created
node/kubemaster untainted
#####
##### Create Kubernetes Master Node Security Key
#####
security key를 생성하였습니다.
#####
##### Input your Kubernetes Dashboard Password
#####
인증서 비밀번호를 입력하세요 :
Enter Export Password:
Verifying - Enter Export Password:
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
******************************************** Kubernetes Master Node 설정을 완료하였습니다. **********************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
********************************************* Kubernetes Worker Node 구성을 시작합니다. *************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
#####
##### Check the SwapMemory
#####
swap 영역을 해제 하였습니다.
#####
##### Confirm Master Join info
#####
kubeadm join 192.168.56.102:6443 --token bos3m9.0ncjrroq0i6pnge7 --discovery-token-ca-cert-hash sha256:8e8495ef0096544648fb5cc6d023f279e336c2a21d56e55be62af7167448989c
#####
##### Join Kubernetes Master Node
#####
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
Trying to pull repository docker.io/nara0617/my-apache2 ...
latest: Pulling from docker.io/nara0617/my-apache2
Digest: sha256:f3b7575cb17091534b8916f49aa373c4e1f469338facc231e50c4b3a0aca0a9a
Status: Image is up to date for docker.io/nara0617/my-apache2:latest
#####
##### Configuration Kubernetes Deployment, Pod, ReplicaSet
#####
deployment.apps/apache-docker-deployment created
#####
##### Configuration Kubernetes Service
#####
service/apache-docker-service created
#####
##### Configuration Kubernetes Ingress Controller
#####
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created
#####
##### Configuration Kubernetes Ingress, Role & Domain
#####
외부에서 접속할 도메인을 입력하세요 :
test.nrson.co.kr
ingress.extensions/nginx-ingress created
#####
##### Configuration Kubernetes External IP
#####
service/ingress-nginx created
#####
##### Delete temporary files
#####
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
******************************************* Kubernetes Worker Node 구성을 완료하였습니다. ***********************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
*****************************************************************************************************************************************
README
=========================================================================================================================================
== 모든 구성이 완료되었습니다. ==========================================================================================================
== 구성을 종료합니다. ===================================================================================================================
== ~/encrypt/ 하위의 ca.crt, kubecfg.p12, token.txt 파일을 대시보드에 접속하기 원하는 로컬에 다운로드 받으세요. =========================
== 개인키 등록 : C:\Windows\System32>certutil.exe -addstore Root [파일 경로]\ca.crt ===================================================
== 루트 인증 기관 등록 : certutil.exe -p dhfhskalsC! -user -importPFX [파일 경로]\kubecfg.p12 ===========================================
== Kubernetes Dashboard에 접속하여 구성 정보를 확인하시고, ==============================================================================
== DashboardUrl (https://[master_ip]:6443/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login) ==
== 테스트 페이지에 접속해 보세요. =======================================================================================================
== TestUrl (http://[DOMAIN_URL]/index.html) / Worker Node IP를 Hosts File에 등록하세요. ========================================
=========================================================================================================================================
Kubernetes 기동 및 설정 시간이 총 118seconds 소요되었습니다.
[root@kubemaster kubernetes_onestepboot]#위와 같이 수행되며, 실행 과정에 몇가지 트러블 슈팅 포인트를 살펴보자면
a) docker ps -a 확인 시 etcd process가 재기동되는 현상
docker logs [ETCD Container ID] 확인 시 etcdmain: open /etc/kubernetes/pki/etcd/peer.crt: permission denied 에러가 발생할 다음 포스팅을 참고합니다.
[TroubleShooting] etcdmain: open /etc/kubernetes/pki/etcd/peer.crt: permission denied
b) sshpass를 통해 명령을 전송하였으나, 반응이 없을 경우
sshpass -p$WORKER_PASS ssh -o StrictHostKeyChecking=no root@$WORKER_NODE $workerJoin 와 같이 StrictHostKeyChecking=no 옵션을 추가합니다.
c) /var/log/messages 확인 시 err services "ingress-nginx" not found 발생
Aug 18 01:19:57 kubemaster journal: W0817 16:19:57.121256 8 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err services "ingress-nginx" not found 에러가 수없이 반복적으로 출력될 경우 다음 포스팅을 참고합니다.
[TroubleShooting] err services "ingress-nginx" not found
1. Kubernetes 0.kubernetesOneStepInfo.sh로 구동정보 확인하기
두번째로 0.kubernetesOneStepInfo.sh을 이용하여 기동이 완료되었는지 확인해 보도록 하겠습니다.
[root@kubemaster kubernetes_onestepboot]# cat 0.kubernetesOneStepInfo.sh
echo "=========================================================================="
echo "==================Start Kubernetes Info==================================="
echo "=========================================================================="
echo "==================Kubernetes Deployment Info=============================="
kubectl get deployments --all-namespaces
echo "==================Kubernetes Pod Info====================================="
kubectl get pods --all-namespaces
echo "==================Kubernetes Replica Set Info============================="
kubectl get replicaset --all-namespaces
echo "==================Kubernetes Service Info================================="
kubectl get services --all-namespaces
echo "==================Kubernetes Ingress Info================================="
kubectl get ingress --all-namespaces
echo "==================Kubernetes Config Map Info=============================="
kubectl get configmap --all-namespaces
echo "=========================================================================="
echo "==================End Kubernetes Info====================================="
echo "=========================================================================="
[root@kubemaster kubernetes_onestepboot]#Kubernetes가 정상 기동되었는지 여부를 확인하기 위한 명령어 모음 정도로 생각하면 됩니다.
[root@kubemaster kubernetes_onestepboot]# sh 0.kubernetesOneStepInfo.sh
==========================================================================
==================Start Kubernetes Info===================================
==========================================================================
==================Kubernetes Deployment Info==============================
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default apache-docker-deployment 2/2 2 2 18m
ingress-nginx nginx-ingress-controller 1/1 1 1 18m
kube-system coredns 2/2 2 2 19m
kubernetes-dashboard kubernetes-dashboard 1/1 1 1 19m
kubernetes-dashboard kubernetes-metrics-scraper 1/1 1 1 19m
==================Kubernetes Pod Info=====================================
NAMESPACE NAME READY STATUS RESTARTS AGE
default apache-docker-deployment-6c44c89bfb-b9v87 1/1 Running 0 18m
default apache-docker-deployment-6c44c89bfb-lqhwb 1/1 Running 0 18m
ingress-nginx nginx-ingress-controller-6d88996bf-g26rh 1/1 Running 0 18m
kube-system coredns-5c98db65d4-7qz7r 1/1 Running 0 18m
kube-system coredns-5c98db65d4-7ttdr 1/1 Running 0 18m
kube-system etcd-kubemaster 1/1 Running 0 17m
kube-system kube-apiserver-kubemaster 1/1 Running 0 17m
kube-system kube-controller-manager-kubemaster 1/1 Running 0 18m
kube-system kube-proxy-rfd9b 1/1 Running 0 18m
kube-system kube-proxy-wxxhz 1/1 Running 0 18m
kube-system kube-scheduler-kubemaster 1/1 Running 0 17m
kube-system weave-net-hcmf7 2/2 Running 0 18m
kube-system weave-net-qrxgf 2/2 Running 0 18m
kubernetes-dashboard kubernetes-dashboard-67f4bff96d-fg9cl 1/1 Running 0 18m
kubernetes-dashboard kubernetes-metrics-scraper-86456cdd8f-ptfnd 1/1 Running 0 18m
==================Kubernetes Replica Set Info=============================
NAMESPACE NAME DESIRED CURRENT READY AGE
default apache-docker-deployment-6c44c89bfb 2 2 2 18m
ingress-nginx nginx-ingress-controller-6d88996bf 1 1 1 18m
kube-system coredns-5c98db65d4 2 2 2 18m
kubernetes-dashboard kubernetes-dashboard-67f4bff96d 1 1 1 18m
kubernetes-dashboard kubernetes-metrics-scraper-86456cdd8f 1 1 1 18m
==================Kubernetes Service Info=================================
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default apache-docker-service ClusterIP 10.103.14.107 <none> 80/TCP 18m
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 19m
ingress-nginx ingress-nginx ClusterIP 10.97.228.146 192.168.56.102,192.168.56.101 80/TCP 17m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 19m
kubernetes-dashboard dashboard-metrics-scraper ClusterIP 10.103.160.52 <none> 8000/TCP 19m
kubernetes-dashboard kubernetes-dashboard ClusterIP 10.103.137.145 <none> 443/TCP 19m
==================Kubernetes Ingress Info=================================
NAMESPACE NAME HOSTS ADDRESS PORTS AGE
default nginx-ingress test.nrson.co.kr 192.168.56.101,192.168.56.102 80 17m
==================Kubernetes Config Map Info==============================
NAMESPACE NAME DATA AGE
ingress-nginx ingress-controller-leader-nginx 0 18m
ingress-nginx nginx-configuration 0 18m
ingress-nginx tcp-services 0 18m
ingress-nginx udp-services 0 18m
kube-public cluster-info 2 19m
kube-system coredns 1 19m
kube-system extension-apiserver-authentication 6 19m
kube-system kube-proxy 2 19m
kube-system kubeadm-config 2 19m
kube-system kubelet-config-1.15 1 19m
kube-system weave-net 0 18m
kubernetes-dashboard kubernetes-dashboard-settings 0 19m
==========================================================================
==================End Kubernetes Info=====================================
==========================================================================
[root@kubemaster kubernetes_onestepboot]# 정상적으로 기동되었다면 위 수준의 정보가 출력되어야 합니다.
2. Kubernetes 0.kubernetesOneStepDown.sh로 Kubernetes 다운하기
마지막으로 0.kubernetesOneStepDown.sh을 이용하여 K8S를 내립니다.
[root@kubemaster kubernetes_onestepboot]# sh 0.kubernetesOneStepDown.sh
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
W0818 01:42:04.663505 15489 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.
The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
[reset] Removing info for node "kubemaster" from the ConfigMap "kubeadm-config" in the "kube-system" Namespace
W0818 01:42:05.956989 24316 removeetcdmember.go:61] [reset] failed to remove etcd member: error syncing endpoints with etc: etcdclient: no available endpoints
.Please manually remove this etcd member using etcdctl
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.
The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
[root@kubemaster kubernetes_onestepboot]# sh 0.kubernetesOneStepInfo.sh
==========================================================================
==================Start Kubernetes Info===================================
==========================================================================
==================Kubernetes Deployment Info==============================
The connection to the server 192.168.56.102:6443 was refused - did you specify the right host or port?
==================Kubernetes Pod Info=====================================
The connection to the server 192.168.56.102:6443 was refused - did you specify the right host or port?
==================Kubernetes Replica Set Info=============================
The connection to the server 192.168.56.102:6443 was refused - did you specify the right host or port?
==================Kubernetes Service Info=================================
The connection to the server 192.168.56.102:6443 was refused - did you specify the right host or port?
==================Kubernetes Ingress Info=================================
The connection to the server 192.168.56.102:6443 was refused - did you specify the right host or port?
==================Kubernetes Config Map Info==============================
The connection to the server 192.168.56.102:6443 was refused - did you specify the right host or port?
==========================================================================
==================End Kubernetes Info=====================================
==========================================================================
[root@kubemaster kubernetes_onestepboot]# 쓰다보니 내용이 길어졌는데 요약하자면 다음과 같습니다.
1) Shell Script 실행을 위한 이동 (cd $GIT_REPO_HOME/kubernetesonestepboot/kubernetes_onestepboot/)
2) Shell Script 실행 (sh 0.kubernetesOneStepBootByInternet.sh)
3) Shell Script로 기동 상태 확인 (sh 0.kubernetesOneStepInfo.sh)
4) Shell Script로 프로세스 다운 (sh 0.kubernetesOneStepDown.sh)
정말 간단하죠?
그럼 총 정리를 해보도록 하겠습니다.
지난 포스팅에서 살펴보았던 Kubernetes 설치 과정은
1) Virtual Box 및 CentOS 설치
2) git 설치 (yum install -y git)
3) git repository 다운로드 (git clone https://github.com/sonnaraon/kubernetesonestepboot.git)
4) Shell Script 실행을 위한 이동 (cd $GIT_REPO_HOME/kubernetesonestepboot/kubernetes_onestepinstall/)
5) Shell Script 실행 (sh 1.kubernetesOneStepInstallByInternet.sh)
+
Kubernetes 설정/기동 및 확인 과정은
1) Shell Script 실행을 위한 이동 (cd $GIT_REPO_HOME/kubernetesonestepboot/kubernetes_onestepboot/)
2) Shell Script 실행 (sh 0.kubernetesOneStepBootByInternet.sh)
3) Shell Script로 기동 상태 확인 (sh 0.kubernetesOneStepInfo.sh)
4) Shell Script로 프로세스 다운 (sh 0.kubernetesOneStepDown.sh)
로서 손쉽게 Kubernetes Project를 구성하고 운영할 수 있습니다.
물론 제공된 Yaml 파일을 잘 편집하여 본인이 원하는 새로운 Docker 이미지 환경으로 사용하셔도 무방합니다.
Kubernetes Dashboard 및 Application 호출
마지막으로 지금까지 구성한 Kubernetes Dashboard 및 Application 정보를 확인하고 호출해 보도록 하겠습니다.
1) ExternalIP로 직접 호출한 경우
화면이 위와 같이 404 Not Found와 함께 nginx version이 출력됩니다.
현재 Ingress Nginx에 적용된 Ingress Rule에 domain은 test.nrson.co.kr로 되어 있기 때문에 ip로는 직접 접근이 불가능합니다.
2) Ingress에 등록한 Domain으로 호출한 경우
당연하겠지만, Domain 정보를 hosts 파일에 등록해야하며, 위와 같이 정상적으로 출력되는 것을 확인할 수 있습니다.
3) Kubernetes Dashboard
위와 같이 Workload를 통해 생성된 apache-docker-deployment pod 2개와 각각의 정보들을 왼쪽 메뉴바를 통해 확인할 수 있습니다.
마지막으로 ingress-nginx의 Namespace 정보입니다.
위와 같이 nginx-ingress-controller 정보와 각종 룰 정보들을 확인할 수 있습니다.
포스팅이 굉장히 길어지긴 했지만, 이와 같은 과정으로 Kubernetes를 손쉽게 구축해 볼 수 있도록 가이드를 작성해 보았습니다.
차근차근 곱씹어 보면서 yaml 파일이나 Shell Script도 직접 수정해 보고 이것 저것 테스트 해보면서 스킬업 해 나가시기 바랍니다.
728x90
반응형
'③ 클라우드 > ⓚ Kubernetes' 카테고리의 다른 글
최근에 올라온 글
최근에 달린 댓글
- Total
- Today
- Yesterday
링크
TAG
- JBoss
- JEUS7
- aa
- webtob
- Docker
- SWA
- jeus
- k8s
- 마이크로서비스 아키텍처
- API Gateway
- Architecture
- OpenStack
- MSA
- SA
- 아키텍처
- JEUS6
- node.js
- 오픈스택
- TA
- wildfly
- aws
- Da
- openstack tenant
- apache
- git
- kubernetes
- nodejs
- 쿠버네티스
- openstack token issue
- 마이크로서비스
| 일 | 월 | 화 | 수 | 목 | 금 | 토 |
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | |
| 7 | 8 | 9 | 10 | 11 | 12 | 13 |
| 14 | 15 | 16 | 17 | 18 | 19 | 20 |
| 21 | 22 | 23 | 24 | 25 | 26 | 27 |
| 28 | 29 | 30 |
글 보관함