Kubernetes - Kuberspray Private Registry 적용하기

지금부터 살펴볼 내용은 Kubernetes를 설치하는 다양한 방법 중 Kubespray를 사용하여 구성 시 Docker Private Registry를 어떻게 구성할 것인가에 대한 내용입니다.

Kubernetes를 설치하기 전 Kubespray를 폐쇄망 환경에서는 어떻게 구축해야 하는지에 대한 고찰을 먼저 진행해 보겠습니다.

Kubernetes에서 가이드하는 Kubespray 설치 가이드는 다음과 같습니다.

https://kubernetes.io/docs/setup/production-environment/tools/kubespray/#creating-a-cluster

Installing Kubernetes with Kubespray

Kubespray는 Ansible을 통해 구성하게 되는데, 이때 사용하게 되는 Terraform script를 제공합니다.

해당 URL은 다음과 같습니다.

https://github.com/kubernetes-sigs/kubespray

kubernetes-sigs/kubespray

위 과정을 통해 Kubespray로 Kubernetes를 구성하는 가이드는 주말에 업로드 될 예정입니다.

대부분의 환경은 인터넷 사용이 가능한 환경을 기준으로 작성하게 되지만, 때때로 인터넷 사용이 제약적인 폐쇄망 환경에서 대용량 Kubernetes를 구축해야 하는 경우가 있습니다.

이 경우 유용하게 사용할 수 있을 듯합니다.

Kubespray github 사이트에서는 다음과 같이 Offline 환경에서 구성이 가능하다고 명시하고 있습니다.

Offline environment
In case your servers don't have access to internet (for example when deploying on premises with security constraints),
you'll have, first, to setup the appropriate proxies/caches/mirrors and/or internal repositories and registries and, then,
adapt the following variables to fit your environment before deploying:

- At least foo_image_repo and foo_download_url as described before
(i.e. in case of use of proxies to registries and binaries repositories, checksums and versions do not necessarily need to be changed).
NOTE: Regarding foo_image_repo, when using insecure registries/proxies, you will certainly have to append them to the docker_insecure_registries variable in group_vars/all/docker.yml
- pyrepo_index (and optionally pyrepo_cert)
- Depending on the container_manager
  When container_manager=docker, docker_foo_repo_base_url, docker_foo_repo_gpgkey, dockerproject_bar_repo_base_url and dockerproject_bar_repo_gpgkey
  (where foo is the distribution and bar is system package manager)
  When container_manager=crio, crio_rhel_repo_base_url
- When using Helm, helm_stable_repo_url

위와 같이 docker private registry를 사용하기 위해서는 크게 두가지 파일을 변경해야 합니다.

main.yml

먼저 변경할 파일은 main.yml 파일입니다.

파일 경로 : $KUBESPRAY_HOME/roles/download/defaults/main.yml

[root@nrson kubespray]# cat ./roles/download/defaults/main.yml
---
local_release_dir: /tmp/releases
download_cache_dir: /tmp/kubespray_cache

# do not delete remote cache files after using them
# NOTE: Setting this parameter to TRUE is only really useful when developing kubespray
download_keep_remote_cache: false

# Only useful when download_run_once is false: Localy cached files and images are
# uploaded to kubernetes nodes. Also, images downloaded on those nodes are copied
# back to the ansible runner's cache, if they are not yet preset.
download_force_cache: false

# Used to only evaluate vars from download role
skip_downloads: false

# Optionally skip kubeadm images download
skip_kubeadm_images: false
kubeadm_images: {}

# if this is set to true will only download files once. Doesn't work
# on Container Linux by CoreOS unless the download_localhost is true and localhost
# is running another OS type. Default compress level is 1 (fastest).
download_run_once: False
download_compress: 1

# if this is set to true will download container
download_container: True

# if this is set to true, uses the localhost for download_run_once mode
# (requires docker and sudo to access docker). You may want this option for
# local caching of docker images or for Container Linux by CoreOS cluster nodes.
# Otherwise, uses the first node in the kube-master group to store images
# in the download_run_once mode.
download_localhost: False

# Always pull images if set to True. Otherwise check by the repo's tag/digest.
download_always_pull: False

# Some problems may occur when downloading files over https proxy due to ansible bug
# https://github.com/ansible/ansible/issues/32750. Set this variable to False to disable
# SSL validation of get_url module. Note that kubespray will still be performing checksum validation.
download_validate_certs: True

# Use the first kube-master if download_localhost is not set
download_delegate: "{% if download_localhost %}localhost{% else %}{{ groups['kube-master'][0] }}{% endif %}"

# Arch of Docker images and needed packages
image_arch: "{{host_architecture | default('amd64')}}"

# Versions
kube_version: v1.16.0
kubeadm_version: "{{ kube_version }}"
etcd_version: v3.3.10

# gcr and kubernetes image repo define
gcr_image_repo: "192.168.56.107:12000"
kube_image_repo: "{{ gcr_image_repo }}/google-containers"

# docker image repo define
docker_image_repo: "192.168.56.107:12000"

# quay image repo define
quay_image_repo: "192.168.56.107:12000"

# TODO(mattymo): Move calico versions to roles/network_plugins/calico/defaults
# after migration to container download
calico_version: "v3.7.3"
calico_ctl_version: "v3.7.3"
calico_cni_version: "v3.7.3"
calico_policy_version: "v3.7.3"
calico_rr_version: "v0.6.1"
calico_typha_version: "v3.7.3"

flannel_version: "v0.11.0"
flannel_cni_version: "v0.3.0"

cni_version: "v0.8.1"

weave_version: 2.5.2
pod_infra_version: 3.1
contiv_version: 1.2.1
cilium_version: "v1.5.5"
kube_ovn_version: "v0.6.0"
kube_router_version: "v0.2.5"
multus_version: "v3.2.1"

crictl_version: "v1.16.0"

# Download URLs
kubeadm_download_url: "https://storage.googleapis.com/kubernetes-release/release/{{ kubeadm_version }}/bin/linux/{{ image_arch }}/kubeadm"
hyperkube_download_url: "https://storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/hyperkube"
etcd_download_url: "https://github.com/coreos/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"
cni_download_url: "https://github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
calicoctl_download_url: "https://github.com/projectcalico/calicoctl/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
crictl_download_url: "https://github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"

crictl_checksums:
  arm:
    v1.16.0: 331c49bd9196009b8230f7a36ec272924a7bcf4c1614ecddf0eb9598c787da0e
    v1.15.0: f31f8c3b4791608a48d030d1aa1a694a73849ae057b23a90ce4ef17e5afde9e8
    v1.14.0: 9910cecfd6558239ba015323066c7233d8371af359b9ddd0b2a35d5223bcf945
  arm64:
    v1.16.0: aa118c31d6f6fd2d24bb2de4a33598a14a5952e1d01f93d5c3267c2b5334743b
    v1.15.0: 785c3da7e058f6fd00b0a48de24b9199eb6bae940d13f509c44ea6dd7ad9ffcd
    v1.14.0: f76b3d00a272c8d210e9a45f77d07d3770bee310d99c4fd9a72d6f55278882e5
  amd64:
    v1.16.0: a3eefa10a483c643ad85aee3d7832a720976ef7e80dde46b212eaaacd7d09512
    v1.15.0: c3b71be1f363e16078b51334967348aab4f72f46ef64a61fe7754e029779d45a
    v1.14.0: 483c90a9fe679590df4332ba807991c49232e8cd326c307c575ecef7fe22327b

# Checksums
hyperkube_checksums:
  arm:
    v1.16.0: 4f2e4ffcf7b7f40c70c637c7be9dd51cd1ee29763696011149f315e90339330b
    v1.15.3: 100d8bddb29e77397b90e6dfbcf0af2d901a90ea4bde90b83b5a39f394c3900b
    v1.15.2: eeaa8e071541c7bcaa186ff1d2919d076b27ef70c9e9df70f910756eba55dc99
    v1.15.1: fc5af96fd9341776d84c38675be7b8045dee20af327af9331972c422a4109918
    v1.15.0: d923c781031bfd97d0fbe50311e4d7c3616aa5b6d466b99049931f09d73d07b9
  arm64:
    v1.16.0: 0431fff5b32042369a3f233afb7b3adcda7ae5446f31700819ed986f3624bc69
    v1.15.3: 1e3e70b8d1e8ebc642f2801d9c7938a27764dfb2f5aea432ab4326d43c04a1f5
    v1.15.2: c4cf69f52c7013faee9d54e0f376e0732a4a7b0f7ffc7241e9b7e28bad0ac77f
    v1.15.1: 80ed372c5f6c5178df88616175310057c06bdc9d0905953814a1927eb3aaa657
    v1.15.0: 824af7d925b87a5ade63575b98b59ee81005fc76eac1dc399602308d7a60bc3c
  amd64:
    v1.16.0: 00b54ca779db1749ed714b19bb2b9a0333b39048af134f9199e4a5441c1b8324
    v1.15.3: 3685c65b4fb85d552f77346900affc2e9a1bc997b4cd3dde0e705fd8c1d9be7a
    v1.15.2: ab885606438748eb89a7738e219f5353d94c40c63a4935a539ce89760280f065
    v1.15.1: 22b7b1e7f5f2a452d62e0ca4c2cba67119c51e04219aaeaf8452825f9177069e
    v1.15.0: 3cc72cc58517b97c608c7a59a20255675bc70f07217c9e11e58cac7746139283
kubeadm_checksums:
  arm:
    v1.16.0: 6c666958e11b7d4513adecb3107c885c98bdc79f38d369c9f80eaaeae4ddfe66
    v1.15.3: 6c6fa56810908b5be83882094ea199844edc94b7e969160623c86512d9251c06
    v1.15.2: 4b35ad0031c08a83de7c8d9f9bbed6a30d93a5c74e16ea9e6211ad2e0e12bdd1
    v1.15.1: 855abd520291dcef0577a1a2ef87a70f522fd2b22603a12abcd86c2f7ec9c022
    v1.15.0: 9464030a1d4e101de5f47348f3514d5a9eb95cbce2e5e31f53ada1ca485cf75e
  arm64:
    v1.16.0: 9a1d21bfb6bd15697ac010665e5917a5364b340d5b60f2f0302c179d75da0f3f
    v1.15.3: 6f472bc8ab1ba3d76448bd45b200edef96741e5affde8dc1429300af3a4904d8
    v1.15.2: d3b6ee2048b366726ca366d2db4c46b2cacc38e8ec09cc35781d16593753d930
    v1.15.1: 44fbfad0f1026d249fc4f365f1e9562cd52d75360d4d1032731122ba5a4d57dc
    v1.15.0: fe3c79070814fe847a23209b1027672fe5c5e7e5c9611e329225058926836f96
  amd64:
    v1.16.0: 18f30d65fb05148c73cc07c77a83f4a2427379af493ca9f60eda42239409e7ef
    v1.15.3: ec56a00bc8d9ec4ac2b081a3b2127d8593daf3b2c86560cf9e6cba5ada2d5a80
    v1.15.2: fe2a13a1dea73249560ea44ab54c0359a9722e9c66832f6bcad86798438cba2f
    v1.15.1: 3d42441ae177826f1181e559cd2a729464ca8efadef196cfa0e8053a615333b5
    v1.15.0: fc4aa44b96dc143d7c3062124e25fed671cab884ebb8b2446edd10abb45e88c2

etcd_binary_checksums:
  # Etcd does not have arm32 builds at the moment, having some dummy value is
  # required to avoid "no attribute" error
  arm: 0
  arm64: 5ec97b0b872adce275b8130d19db314f7f2b803aeb24c4aae17a19e2d66853c4
  amd64: 1620a59150ec0a0124a65540e23891243feb2d9a628092fb1edcc23974724a45
cni_binary_checksums:
  arm: ae6ddbd87c05a79aceb92e1c8c32d11e302f6fc55045f87f6a3ea7e0268b2fda
  arm64: acde854e3def3c776c532ae521c19d8784534918cc56449ff16945a2909bff6d
  amd64: e9bfc78acd3ae71be77eb8f3e890cc9078a33cc3797703b8ff2fc3077a232252
calicoctl_binary_checksums:
  arm:
    v3.6.1: 0
    v3.5.4: 0
    v3.4.4: 0
    v3.7.3: 0
  amd64:
    v3.6.1: 3b01336de37550e020343d62a38c96c4605d33a3ed7ddba2fe38bc172a5b42b5
    v3.5.4: 197194b838cc2a9a7455c2ebd5505a5e24f8f3d994eb75c17f5dd568944100b8
    v3.4.4: 93bd084e053cf1bf3b7fef369677bd6767c30fe7135e2c7e044e31693422ef61
    v3.7.3: 932f68e893e80e95e10f064f1e7745e438d456f41a6ff12d11bb16ca0cab735c
  arm64:
    v3.6.1: 60fbaeb257061647bdf12b5ede7a0d4298a5ee216f6472e5a92bb14ef5c2a5d3
    v3.5.4: a4481178665658658a73e4ceca9a1dff5cccded4179615c91d1c3e49fd96f237
    v3.4.4: ff35d9e8b5c00e9fe47d05e8f5123ec98fd641370f8cd93f4fbb3d913da77ab6
    v3.7.3: 7cfaab25c287f7ef93b2682d060b55bf39f76b668540de50376b5ed174209832

etcd_binary_checksum: "{{ etcd_binary_checksums[image_arch] }}"
cni_binary_checksum: "{{ cni_binary_checksums[image_arch] }}"
hyperkube_binary_checksum: "{{ hyperkube_checksums[image_arch][kube_version] }}"
kubeadm_binary_checksum: "{{ kubeadm_checksums[image_arch][kubeadm_version] }}"
calicoctl_binary_checksum: "{{ calicoctl_binary_checksums[image_arch][calico_ctl_version] }}"
crictl_binary_checksum: "{{ crictl_checksums[image_arch][crictl_version] }}"

# Containers
# In some cases, we need a way to set --registry-mirror or --insecure-registry for docker,
# it helps a lot for local private development or bare metal environment.
# So you need define --registry-mirror or --insecure-registry, and modify the following url address.
# example:
# You need to deploy kubernetes cluster on local private development.
# Also provide the address of your own private registry.
# And use --insecure-registry options for docker
etcd_image_repo: "{{ quay_image_repo }}/coreos/etcd"
etcd_image_tag: "{{ etcd_version }}{%- if image_arch != 'amd64' -%}-{{ image_arch }}{%- endif -%}"
flannel_image_repo: "{{ quay_image_repo }}/coreos/flannel"
flannel_image_tag: "{{ flannel_version }}"
flannel_cni_image_repo: "{{ quay_image_repo }}/coreos/flannel-cni"
flannel_cni_image_tag: "{{ flannel_cni_version }}"
calico_node_image_repo: "{{ docker_image_repo }}/calico/node"
calico_node_image_tag: "{{ calico_version }}"
calico_cni_image_repo: "{{ docker_image_repo }}/calico/cni"
calico_cni_image_tag: "{{ calico_cni_version }}"
calico_policy_image_repo: "{{ docker_image_repo }}/calico/kube-controllers"
calico_policy_image_tag: "{{ calico_policy_version }}"
calico_rr_image_repo: "{{ docker_image_repo }}/calico/routereflector"
calico_rr_image_tag: "{{ calico_rr_version }}"
calico_typha_image_repo: "{{ docker_image_repo }}/calico/typha"
calico_typha_image_tag: "{{ calico_typha_version }}"
pod_infra_image_repo: "{{ gcr_image_repo }}/google_containers/pause-{{ image_arch }}"
pod_infra_image_tag: "{{ pod_infra_version }}"
install_socat_image_repo: "{{ docker_image_repo }}/xueshanf/install-socat"
install_socat_image_tag: "latest"
netcheck_version: "v1.0"
netcheck_agent_image_repo: "{{ quay_image_repo }}/l23network/k8s-netchecker-agent"
netcheck_agent_image_tag: "{{ netcheck_version }}"
netcheck_server_image_repo: "{{ quay_image_repo }}/l23network/k8s-netchecker-server"
netcheck_server_image_tag: "{{ netcheck_version }}"
weave_kube_image_repo: "{{ docker_image_repo }}/weaveworks/weave-kube"
weave_kube_image_tag: "{{ weave_version }}"
weave_npc_image_repo: "{{ docker_image_repo }}/weaveworks/weave-npc"
weave_npc_image_tag: "{{ weave_version }}"
contiv_image_repo: "{{ docker_image_repo }}/contiv/netplugin"
contiv_image_tag: "{{ contiv_version }}"
contiv_init_image_repo: "{{ docker_image_repo }}/contiv/netplugin-init"
contiv_init_image_tag: "latest"
contiv_auth_proxy_image_repo: "{{ docker_image_repo }}/contiv/auth_proxy"
contiv_auth_proxy_image_tag: "{{ contiv_version }}"
contiv_etcd_init_image_repo: "{{ docker_image_repo }}/ferest/etcd-initer"
contiv_etcd_init_image_tag: latest
contiv_ovs_image_repo: "{{ docker_image_repo }}/contiv/ovs"
contiv_ovs_image_tag: "latest"
cilium_image_repo: "{{ docker_image_repo }}/cilium/cilium"
cilium_image_tag: "{{ cilium_version }}"
cilium_init_image_repo: "{{ docker_image_repo }}/cilium/cilium-init"
cilium_init_image_tag: "2019-04-05"
cilium_operator_image_repo: "{{ docker_image_repo }}/cilium/operator"
cilium_operator_image_tag: "{{ cilium_version }}"
kube_ovn_db_image_repo: "{{ docker_image_repo }}/kubeovn/kube-ovn-db"
kube_ovn_node_image_repo: "{{ docker_image_repo }}/kubeovn/kube-ovn-node"
kube_ovn_cni_image_repo: "{{ docker_image_repo }}/kubeovn/kube-ovn-cni"
kube_ovn_controller_image_repo: "{{ docker_image_repo }}/kubeovn/kube-ovn-controller"
kube_ovn_db_image_tag: "{{ kube_ovn_version }}"
kube_ovn_node_image_tag: "{{ kube_ovn_version }}"
kube_ovn_controller_image_tag: "{{ kube_ovn_version }}"
kube_ovn_cni_image_tag: "{{ kube_ovn_version }}"
kube_router_image_repo: "{{ docker_image_repo }}/cloudnativelabs/kube-router"
kube_router_image_tag: "{{ kube_router_version }}"
multus_image_repo: "{{ docker_image_repo }}/nfvpe/multus"
multus_image_tag: "{{ multus_version }}"
nginx_image_repo: "{{ docker_image_repo }}/library/nginx"
nginx_image_tag: 1.15

haproxy_image_repo: "{{ docker_image_repo }}/library/haproxy"
haproxy_image_tag: 1.9

coredns_version: "1.6.0"
coredns_image_repo: "{{ docker_image_repo }}/coredns/coredns"
coredns_image_tag: "{{ coredns_version }}"

nodelocaldns_version: "1.15.5"
nodelocaldns_image_repo: "{{ kube_image_repo }}/k8s-dns-node-cache"
nodelocaldns_image_tag: "{{ nodelocaldns_version }}"

dnsautoscaler_version: 1.6.0
dnsautoscaler_image_repo: "{{ kube_image_repo }}/cluster-proportional-autoscaler-{{ image_arch }}"
dnsautoscaler_image_tag: "{{ dnsautoscaler_version }}"
test_image_repo: "{{ docker_image_repo }}/library/busybox"
test_image_tag: latest
busybox_image_repo: "{{ docker_image_repo }}/library/busybox"
busybox_image_tag: 1.29.2
helm_version: "v2.14.3"
helm_image_repo: "{{ docker_image_repo }}/lachlanevenson/k8s-helm"
helm_image_tag: "{{ helm_version }}"
tiller_image_repo: "{{ gcr_image_repo }}/kubernetes-helm/tiller"
tiller_image_tag: "{{ helm_version }}"

registry_image_repo: "{{ docker_image_repo }}/library/registry"
registry_image_tag: "2.6"
registry_proxy_image_repo: "{{ gcr_image_repo }}/google_containers/kube-registry-proxy"
registry_proxy_image_tag: "0.4"
metrics_server_version: "v0.3.3"
metrics_server_image_repo: "{{ gcr_image_repo }}/google_containers/metrics-server-amd64"
metrics_server_image_tag: "{{ metrics_server_version }}"
local_volume_provisioner_image_repo: "{{ quay_image_repo }}/external_storage/local-volume-provisioner"
local_volume_provisioner_image_tag: "v2.3.2"
cephfs_provisioner_image_repo: "{{ quay_image_repo }}/external_storage/cephfs-provisioner"
cephfs_provisioner_image_tag: "v2.1.0-k8s1.11"
rbd_provisioner_image_repo: "{{ quay_image_repo }}/external_storage/rbd-provisioner"
rbd_provisioner_image_tag: "v2.1.1-k8s1.11"
local_path_provisioner_image_repo: "{{ docker_image_repo }}/rancher/local-path-provisioner"
local_path_provisioner_image_tag: "v0.0.2"
ingress_nginx_controller_image_repo: "{{ quay_image_repo }}/kubernetes-ingress-controller/nginx-ingress-controller"
ingress_nginx_controller_image_tag: "0.25.1"
cert_manager_version: "v0.5.2"
cert_manager_controller_image_repo: "{{ quay_image_repo }}/jetstack/cert-manager-controller"
cert_manager_controller_image_tag: "{{ cert_manager_version }}"
addon_resizer_version: "1.8.3"
addon_resizer_image_repo: "{{ kube_image_repo }}/addon-resizer"
addon_resizer_image_tag: "{{ addon_resizer_version }}"

dashboard_image_repo: "{{ gcr_image_repo }}/google_containers/kubernetes-dashboard-{{ image_arch }}"
dashboard_image_tag: "v1.10.1"

image_pull_command: "{{ docker_bin_dir }}/docker pull"
image_info_command: "{{ docker_bin_dir }}/docker images -q | xargs {{ docker_bin_dir }}/docker inspect -f \"{{ '{{' }} if .RepoTags {{ '}}' }}{{ '{{' }} (index .RepoTags 0) {{ '}}' }}{{ '{{' }} end {{ '}}' }}{{ '{{' }} if .RepoDigests {{ '}}' }},{{ '{{' }} (index .RepoDigests 0) {{ '}}' }}{{ '{{' }} end {{ '}}' }}\" | tr '\n' ','"

downloads:
  netcheck_server:
    enabled: "{{ deploy_netchecker }}"
    container: true
    repo: "{{ netcheck_server_image_repo }}"
    tag: "{{ netcheck_server_image_tag }}"
    sha256: "{{ netcheck_server_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  netcheck_agent:
    enabled: "{{ deploy_netchecker }}"
    container: true
    repo: "{{ netcheck_agent_image_repo }}"
    tag: "{{ netcheck_agent_image_tag }}"
    sha256: "{{ netcheck_agent_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  etcd:
    container: "{{ etcd_deployment_type != 'host' }}"
    file: "{{ etcd_deployment_type == 'host' or etcd_kubeadm_enabled }}"
    enabled: true
    version: "{{ etcd_version }}"
    dest: "{{local_release_dir}}/etcd-{{ etcd_version }}-linux-amd64.tar.gz"
    repo: "{{ etcd_image_repo }}"
    tag: "{{ etcd_image_tag }}"
    sha256: >-
     {{ etcd_binary_checksum if (etcd_deployment_type == 'host' or etcd_kubeadm_enabled)
     else etcd_digest_checksum|d(None) }}
    url: "{{ etcd_download_url }}"
    unarchive: true
    owner: "root"
    mode: "0755"
    groups:
      - etcd

  cni:
    enabled: true
    file: true
    version: "{{ cni_version }}"
    dest: "{{local_release_dir}}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
    sha256: "{{ cni_binary_checksum }}"
    url: "{{ cni_download_url }}"
    unarchive: false
    owner: "root"
    mode: "0755"
    groups:
      - k8s-cluster

  kubeadm:
    enabled: true
    file: true
    version: "{{ kubeadm_version }}"
    dest: "{{ local_release_dir }}/kubeadm-{{ kubeadm_version }}-{{ image_arch }}"
    sha256: "{{ kubeadm_binary_checksum }}"
    url: "{{ kubeadm_download_url }}"
    unarchive: false
    owner: "root"
    mode: "0755"
    groups:
      - k8s-cluster

  hyperkube_file:
    enabled: true
    file: true
    version: "{{ kube_version }}"
    dest: "{{ local_release_dir }}/hyperkube-{{ kube_version }}-{{ image_arch }}"
    sha256: "{{ hyperkube_binary_checksum }}"
    url: "{{ hyperkube_download_url }}"
    unarchive: false
    owner: "root"
    mode: "0755"
    groups:
      - k8s-cluster

  crictl:
    file: true
    enabled: "{{ container_manager in ['crio', 'cri', 'containerd'] }}"
    version: "{{ crictl_version }}"
    dest: "{{local_release_dir}}/crictl-{{ crictl_version }}-linux-{{ image_arch }}.tar.gz"
    sha256: "{{ crictl_binary_checksum }}"
    url: "{{ crictl_download_url }}"
    unarchive: true
    owner: "root"
    mode: "0755"
    groups:
      - k8s-cluster

  cilium:
    enabled: "{{ kube_network_plugin == 'cilium' }}"
    container: true
    repo: "{{ cilium_image_repo }}"
    tag: "{{ cilium_image_tag }}"
    sha256: "{{ cilium_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  cilium_init:
    enabled: "{{ kube_network_plugin == 'cilium' }}"
    container: true
    repo: "{{ cilium_init_image_repo }}"
    tag: "{{ cilium_init_image_tag }}"
    sha256: "{{ cilium_init_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  cilium_operator:
    enabled: "{{ kube_network_plugin == 'cilium' }}"
    container: true
    repo: "{{ cilium_operator_image_repo }}"
    tag: "{{ cilium_operator_image_tag }}"
    sha256: "{{ cilium_operator_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  multus:
    enabled: "{{ kube_network_plugin_multus }}"
    container: true
    repo: "{{ multus_image_repo }}"
    tag: "{{ multus_image_tag }}"
    sha256: "{{ multus_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  flannel:
    enabled: "{{ kube_network_plugin == 'flannel' or kube_network_plugin == 'canal' }}"
    container: true
    repo: "{{ flannel_image_repo }}"
    tag: "{{ flannel_image_tag }}"
    sha256: "{{ flannel_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  flannel_cni:
    enabled: "{{ kube_network_plugin == 'flannel' }}"
    container: true
    repo: "{{ flannel_cni_image_repo }}"
    tag: "{{ flannel_cni_image_tag }}"
    sha256: "{{ flannel_cni_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  calicoctl:
    enabled: "{{ kube_network_plugin == 'calico' or kube_network_plugin == 'canal' }}"
    file: true
    version: "{{ calico_ctl_version }}"
    dest: "{{local_release_dir}}/calicoctl"
    sha256: "{{ calicoctl_binary_checksum }}"
    url: "{{ calicoctl_download_url }}"
    unarchive: false
    owner: "root"
    mode: "0755"
    groups:
      - k8s-cluster

  calico_node:
    enabled: "{{ kube_network_plugin == 'calico' or kube_network_plugin == 'canal' }}"
    container: true
    repo: "{{ calico_node_image_repo }}"
    tag: "{{ calico_node_image_tag }}"
    sha256: "{{ calico_node_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  calico_cni:
    enabled: "{{ kube_network_plugin == 'calico' or kube_network_plugin == 'canal' }}"
    container: true
    repo: "{{ calico_cni_image_repo }}"
    tag: "{{ calico_cni_image_tag }}"
    sha256: "{{ calico_cni_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  calico_policy:
    enabled: "{{ enable_network_policy or kube_network_plugin == 'canal' }}"
    container: true
    repo: "{{ calico_policy_image_repo }}"
    tag: "{{ calico_policy_image_tag }}"
    sha256: "{{ calico_policy_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  calico_rr:
    enabled: "{{ peer_with_calico_rr is defined and peer_with_calico_rr and kube_network_plugin == 'calico' }}"
    container: true
    repo: "{{ calico_rr_image_repo }}"
    tag: "{{ calico_rr_image_tag }}"
    sha256: "{{ calico_rr_digest_checksum|default(None) }}"
    groups:
      - calico-rr

  calico_typha:
    enabled: "{{ typha_enabled == 'calico' }}"
    container: true
    repo: "{{ calico_typha_image_repo }}"
    tag: "{{ calico_typha_image_tag }}"
    sha256: "{{ calico_typha_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  weave_kube:
    enabled: "{{ kube_network_plugin == 'weave' }}"
    container: true
    repo: "{{ weave_kube_image_repo }}"
    tag: "{{ weave_kube_image_tag }}"
    sha256: "{{ weave_kube_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  weave_npc:
    enabled: "{{ kube_network_plugin == 'weave' }}"
    container: true
    repo: "{{ weave_npc_image_repo }}"
    tag: "{{ weave_npc_image_tag }}"
    sha256: "{{ weave_npc_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  contiv:
    enabled: "{{ kube_network_plugin == 'contiv' }}"
    container: true
    repo: "{{ contiv_image_repo }}"
    tag: "{{ contiv_image_tag }}"
    sha256: "{{ contiv_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  contiv_auth_proxy:
    enabled: "{{ kube_network_plugin == 'contiv' }}"
    container: true
    repo: "{{ contiv_auth_proxy_image_repo }}"
    tag: "{{ contiv_auth_proxy_image_tag }}"
    sha256: "{{ contiv_auth_proxy_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  contiv_etcd_init:
    enabled: "{{ kube_network_plugin == 'contiv' }}"
    container: true
    repo: "{{ contiv_etcd_init_image_repo }}"
    tag: "{{ contiv_etcd_init_image_tag }}"
    sha256: "{{ contiv_etcd_init_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  kube_ovn_db:
    enabled: "{{ kube_network_plugin == 'kube-ovn' }}"
    container: true
    repo: "{{ kube_ovn_db_image_repo }}"
    tag: "{{ kube_ovn_db_image_tag }}"
    sha256: "{{ kube_ovn_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  kube_ovn_node:
    enabled: "{{ kube_network_plugin == 'kube-ovn' }}"
    container: true
    repo: "{{ kube_ovn_node_image_repo }}"
    tag: "{{ kube_ovn_node_image_tag }}"
    sha256: "{{ kube_ovn_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  kube_ovn_controller:
    enabled: "{{ kube_network_plugin == 'kube-ovn' }}"
    container: true
    repo: "{{ kube_ovn_controller_image_repo }}"
    tag: "{{ kube_ovn_controller_image_tag }}"
    sha256: "{{ kube_ovn_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  kube_ovn_cni:
    enabled: "{{ kube_network_plugin == 'kube-ovn' }}"
    container: true
    repo: "{{ kube_ovn_cni_image_repo }}"
    tag: "{{ kube_ovn_cni_image_tag }}"
    sha256: "{{ kube_ovn_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  kube_router:
    enabled: "{{ kube_network_plugin == 'kube-router' }}"
    container: true
    repo: "{{ kube_router_image_repo }}"
    tag: "{{ kube_router_image_tag }}"
    sha256: "{{ kube_router_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  pod_infra:
    enabled: true
    container: true
    repo: "{{ pod_infra_image_repo }}"
    tag: "{{ pod_infra_image_tag }}"
    sha256: "{{ pod_infra_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  install_socat:
    enabled: "{{ ansible_os_family in ['CoreOS', 'Container Linux by CoreOS'] }}"
    container: true
    repo: "{{ install_socat_image_repo }}"
    tag: "{{ install_socat_image_tag }}"
    sha256: "{{ install_socat_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  nginx:
    enabled: "{{ loadbalancer_apiserver_localhost and loadbalancer_apiserver_type == 'nginx' }}"
    container: true
    repo: "{{ nginx_image_repo }}"
    tag: "{{ nginx_image_tag }}"
    sha256: "{{ nginx_digest_checksum|default(None) }}"
    groups:
      - kube-node

  haproxy:
    enabled: "{{ loadbalancer_apiserver_localhost and loadbalancer_apiserver_type == 'haproxy' }}"
    container: true
    repo: "{{ haproxy_image_repo }}"
    tag: "{{ haproxy_image_tag }}"
    sha256: "{{ haproxy_digest_checksum|default(None) }}"
    groups:
      - kube-node

  coredns:
    enabled: "{{ dns_mode in ['coredns', 'coredns_dual'] }}"
    container: true
    repo: "{{ coredns_image_repo }}"
    tag: "{{ coredns_image_tag }}"
    sha256: "{{ coredns_digest_checksum|default(None) }}"
    groups:
      - kube-master

  nodelocaldns:
    enabled: "{{ enable_nodelocaldns }}"
    container: true
    repo: "{{ nodelocaldns_image_repo }}"
    tag: "{{ nodelocaldns_image_tag }}"
    sha256: "{{ nodelocaldns_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  dnsautoscaler:
    enabled: "{{ dns_mode in ['coredns', 'coredns_dual'] }}"
    container: true
    repo: "{{ dnsautoscaler_image_repo }}"
    tag: "{{ dnsautoscaler_image_tag }}"
    sha256: "{{ dnsautoscaler_digest_checksum|default(None) }}"
    groups:
      - kube-master

  busybox:
    enabled: "{{ kube_network_plugin in ['kube-router'] }}"
    container: true
    repo: "{{ busybox_image_repo }}"
    tag: "{{ busybox_image_tag }}"
    sha256: "{{ busybox_digest_checksum|default(None) }}"
    groups:
      - k8s-cluster

  testbox:
    enabled: false
    container: true
    repo: "{{ test_image_repo }}"
    tag: "{{ test_image_tag }}"
    sha256: "{{ testbox_digest_checksum|default(None) }}"

  helm:
    enabled: "{{ helm_enabled }}"
    container: true
    repo: "{{ helm_image_repo }}"
    tag: "{{ helm_image_tag }}"
    sha256: "{{ helm_digest_checksum|default(None) }}"
    groups:
      - kube-node

  tiller:
    enabled: "{{ helm_enabled }}"
    container: true
    repo: "{{ tiller_image_repo }}"
    tag: "{{ tiller_image_tag }}"
    sha256: "{{ tiller_digest_checksum|default(None) }}"
    groups:
      - kube-node

  registry:
    enabled: "{{ registry_enabled }}"
    container: true
    repo: "{{ registry_image_repo }}"
    tag: "{{ registry_image_tag }}"
    sha256: "{{ registry_digest_checksum|default(None) }}"
    groups:
      - kube-node

  registry_proxy:
    enabled: "{{ registry_enabled }}"
    container: true
    repo: "{{ registry_proxy_image_repo }}"
    tag: "{{ registry_proxy_image_tag }}"
    sha256: "{{ registry_proxy_digest_checksum|default(None) }}"
    groups:
      - kube-node

  metrics_server:
    enabled: "{{ metrics_server_enabled }}"
    container: true
    repo: "{{ metrics_server_image_repo }}"
    tag: "{{ metrics_server_image_tag }}"
    sha256: "{{ metrics_server_digest_checksum|default(None) }}"
    groups:
      - kube-master

  addon_resizer:
    # Currently addon_resizer is only used by metrics server
    enabled: "{{ metrics_server_enabled }}"
    container: true
    repo: "{{ addon_resizer_image_repo }}"
    tag: "{{ addon_resizer_image_tag }}"
    sha256: "{{ addon_resizer_digest_checksum|default(None) }}"
    groups:
      - kube-master

  local_volume_provisioner:
    enabled: "{{ local_volume_provisioner_enabled }}"
    container: true
    repo: "{{ local_volume_provisioner_image_repo }}"
    tag: "{{ local_volume_provisioner_image_tag }}"
    sha256: "{{ local_volume_provisioner_digest_checksum|default(None) }}"
    groups:
      - kube-node

  cephfs_provisioner:
    enabled: "{{ cephfs_provisioner_enabled }}"
    container: true
    repo: "{{ cephfs_provisioner_image_repo }}"
    tag: "{{ cephfs_provisioner_image_tag }}"
    sha256: "{{ cephfs_provisioner_digest_checksum|default(None) }}"
    groups:
      - kube-node

  rbd_provisioner:
    enabled: "{{ rbd_provisioner_enabled }}"
    container: true
    repo: "{{ rbd_provisioner_image_repo }}"
    tag: "{{ rbd_provisioner_image_tag }}"
    sha256: "{{ rbd_provisioner_digest_checksum|default(None) }}"
    groups:
      - kube-node

  local_path_provisioner:
    enabled: "{{ local_volume_provisioner_enabled }}"
    container: true
    repo: "{{ local_path_provisioner_image_repo }}"
    tag: "{{ local_path_provisioner_image_tag }}"
    sha256: "{{ local_path_provisioner_digest_checksum|default(None) }}"
    groups:
      - kube-node

  ingress_nginx_controller:
    enabled: "{{ ingress_nginx_enabled }}"
    container: true
    repo: "{{ ingress_nginx_controller_image_repo }}"
    tag: "{{ ingress_nginx_controller_image_tag }}"
    sha256: "{{ ingress_nginx_controller_digest_checksum|default(None) }}"
    groups:
      - kube-node

  cert_manager_controller:
    enabled: "{{ cert_manager_enabled }}"
    container: true
    repo: "{{ cert_manager_controller_image_repo }}"
    tag: "{{ cert_manager_controller_image_tag }}"
    sha256: "{{ cert_manager_controller_digest_checksum|default(None) }}"
    groups:
      - kube-node

  dashboard:
    enabled: "{{ dashboard_enabled }}"
    container: true
    repo: "{{ dashboard_image_repo }}"
    tag: "{{ dashboard_image_tag }}"
    sha256: "{{ dashboard_digest_checksum|default(None) }}"
    groups:
      - kube-master

download_defaults:
  container: false
  file: false
  repo: None
  tag: None
  enabled: false
  dest: None
  version: None
  url: None
  unarchive: false
  owner: kube
  mode: None
[root@nrson kubespray]#

환경파일 중 

# gcr and kubernetes image repo define 
gcr_image_repo: "192.168.56.107:12000" 
kube_image_repo: "{{ gcr_image_repo }}/google-containers" 

# docker image repo define 
docker_image_repo: "192.168.56.107:12000" 

# quay image repo define 
quay_image_repo: "192.168.56.107:12000"

위 Docker Registry를 로컬 환경에 구성한 Registry로 변경해 줍니다.

테스트 환경은 앞선 포스팅과 같이 Nexus3 Docker Private Registry를 사용하였으며, 12000번 포트를 기본으로 사용합니다.

docker.yml

다음으로 변경할 파일은 docker.yml 파일입니다.

파일 위치 : $KUBESPRAY_HOME/inventory/sample/group_vars/all/docker.yml

[root@nrson kubespray]# cat ./inventory/sample/group_vars/all/docker.yml
---
## Uncomment this if you want to force overlay/overlay2 as docker storage driver
## Please note that overlay2 is only supported on newer kernels
# docker_storage_options: -s overlay2

## Enable docker_container_storage_setup, it will configure devicemapper driver on Centos7 or RedHat7.
docker_container_storage_setup: false

## It must be define a disk path for docker_container_storage_setup_devs.
## Otherwise docker-storage-setup will be executed incorrectly.
# docker_container_storage_setup_devs: /dev/vdb

## Uncomment this if you have more than 3 nameservers, then we'll only use the first 3.
docker_dns_servers_strict: false

# Path used to store Docker data
docker_daemon_graph: "/var/lib/docker"

## Used to set docker daemon iptables options to true
docker_iptables_enabled: "false"

# Docker log options
# Rotate container stderr/stdout logs at 50m and keep last 5
docker_log_opts: "--log-opt max-size=50m --log-opt max-file=5"

# define docker bin_dir
docker_bin_dir: "/usr/bin"

# keep docker packages after installation; speeds up repeated ansible provisioning runs when '1'
# kubespray deletes the docker package on each run, so caching the package makes sense
docker_rpm_keepcache: 0

## An obvious use case is allowing insecure-registry access to self hosted registries.
## Can be ipaddress and domain_name.
## example define 172.19.16.11 or mirror.registry.io
docker_insecure_registries:
#   - mirror.registry.io
#   - 172.19.16.11
   - 192.168.56.107:12000

## Add other registry,example China registry mirror.
# docker_registry_mirrors:
#   - https://registry.docker-cn.com
#   - https://mirror.aliyuncs.com

## If non-empty will override default system MounFlags value.
## This option takes a mount propagation flag: shared, slave
## or private, which control whether mounts in the file system
## namespace set up for docker will receive or propagate mounts
## and unmounts. Leave empty for system default
# docker_mount_flags:

## A string of extra options to pass to the docker daemon.
## This string should be exactly as you wish it to appear.
docker_options: >-
  {%- if docker_insecure_registries is defined %}
  {{ docker_insecure_registries | map('regex_replace', '^(.*)$', '--insecure-registry=\1' ) | list | join(' ') }}
  {%- endif %}
  {% if docker_registry_mirrors is defined %}
  {{ docker_registry_mirrors | map('regex_replace', '^(.*)$', '--registry-mirror=\1' ) | list | join(' ') }}
  {%- endif %}
  {%- if docker_version != "latest" and docker_version is version('17.05', '<') %}
  --graph={{ docker_daemon_graph }} {% if ansible_os_family not in ["openSUSE Leap", "openSUSE Tumbleweed", "Suse"] %}{{ docker_log_opts }}{% endif %}
  {%- else %}
  --data-root={{ docker_daemon_graph }} {% if ansible_os_family not in ["openSUSE Leap", "openSUSE Tumbleweed", "Suse"] %}{{ docker_log_opts }}{% endif %}
  {%- endif %}
  {%- if ansible_architecture == "aarch64" and ansible_os_family == "RedHat" %}
  --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current
  --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd
  --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --signature-verification=false
  {%- endif -%}
[root@nrson kubespray]#

docker.yml 파일에서는 docker 설정을 변경할 수 있습니다.

먼저 docker registry가 secure 하지 않을 경우 (http를 사용할 경우) insecure_url을 추가하기 위해 아래와 같은 부분을 찾아 추가해 주도록 합니다.

docker_insecure_registries:
#   - mirror.registry.io
#   - 172.19.16.11
   - 192.168.56.107:12000

수정이 완료되면 저장 후 kubespray를 ansible-playbook으로 실행하면, 다음과 같이 Local Registry에서 docker 이미지를 다운로드 받는 것을 확인할 수 있습니다.

[root@nrson kubespray]# docker images
REPOSITORY                                                                    TAG                 IMAGE ID            CREATED             SIZE
192.168.56.107:12000/google-containers/kube-apiserver                          v1.16.0             b305571ca60a        3 weeks ago         217MB
192.168.56.107:12000/google-containers/kube-controller-manager                 v1.16.0             06a629a7e51c        3 weeks ago         163MB
192.168.56.107:12000/google-containers/kube-proxy                              v1.16.0             c21b0c7400f9        3 weeks ago         86.1MB
192.168.56.107:12000/google-containers/kube-scheduler                          v1.16.0             301ddc62b80b        3 weeks ago         87.3MB
192.168.56.107:12000/google-containers/k8s-dns-node-cache                      1.15.5              b477eb2ed326        7 weeks ago         62.5MB
192.168.56.107:12000/coredns/coredns                                           1.6.0               680bc53e5985        2 months ago        42.2MB
192.168.56.107:12000/calico/node                                               v3.7.3              bf4ff15c9db0        4 months ago        156MB
192.168.56.107:12000/calico/cni                                                v3.7.3              1a6ade52d471        4 months ago        135MB
192.168.56.107:12000/calico/kube-controllers                                   v3.7.3              283860d96794        4 months ago        46.8MB
192.168.56.107:12000/library/nginx                                             1.15                53f3fd8007f7        5 months ago        109MB
192.168.56.107:12000/google-containers/cluster-proportional-autoscaler-amd64   1.6.0               dfe4432cd2e2        5 months ago        47.7MB
192.168.56.107:12000/external_storage/local-volume-provisioner                 v2.3.2              c3e26d3f1640        5 months ago        111MB
192.168.56.107:12000/google_containers/kubernetes-dashboard-amd64              v1.10.1             f9aed6605b81        9 months ago        122MB
192.168.56.107:12000/rancher/local-path-provisioner                            v0.0.2              b621dadd3581        12 months ago       327MB
192.168.56.107:12000/coreos/etcd                                               v3.3.10             643c21638c1c        12 months ago       39.5MB
192.168.56.107:12000/google_containers/pause-amd64                             3.1                 da86e6ba6ca1        21 months ago       742kB
192.168.56.107:12000/google-containers/pause                                   3.1                 da86e6ba6ca1        21 months ago       742kB
[root@nrson kubespray]#

이와 같은 방법으로 kubespray를 구성하면, 손쉬운 Kubernetes를 구성함과 동시에 Private 환경에서 적용할 수 있을 것입니다.

그럼 다음 포스팅에서는 직접 Kubespray로 Kubernetes를 구성해 보도록 하겠습니다.