Harbor Docker & Helm Repository 설치 및 구성

Harbor는 Docker Repository와 Helm Chart를 함께 관리할 수 있는 오픈소스 리포지토리이다.

Docker Repository 역할을 기본으로 Clair 기반 도커 이미지 취약점 점검, Chartmuseum 기반 Helm Repository를 함께 제공하여 통합된 클라우드 환경기반 repository를 구축할 수 있다.

본 포스팅에서는 Harbor 구축 과정은 물론, Harbor를 통해 Docker Repository, Docker Image 취약점 분석, Helm Chart Repository 등의 기본 기능 및 활용 가능한 다양한 방안을 모색해 보도록 하자.

 Harbor Install 

Harbor 공식 홈페이지 참조 : https://goharbor.io/docs/2.0.0/install-config/download-installer/

Harbor 공식 GitHub 참조 : https://github.com/goharbor/harbor/releases

 

1. Download Harbor

(curl -s https://api.github.com/repos/goharbor/harbor/releases/latest | grep browser_download_url | cut -d '"' -f 4 | grep '\.tgz$' | wget -qi -)

[root@ciserver ~]# mkdir harbor 
[root@ciserver ~]# cd harbor/ 
[root@ciserver harbor]# curl -s https://api.github.com/repos/goharbor/harbor/releases/latest | grep browser_download_url | cut -d '"' -f 4 | grep '\.tgz$' | wget -qi - 
[root@ciserver harbor]# ls -la 
total 1326036 
drwxr-xr-x.  2 root root       184 Aug  2 04:25 . 
dr-xr-x---. 20 root root      4096 Aug  2 04:23 .. 
-rw-r--r--.  1 root root 678912854 Jul 15 00:27 harbor-offline-installer-v1.10.4.tgz
-rw-r--r--.  1 root root      8484 Jul 15 00:37 harbor-online-installer-v1.10.4.tgz
[root@ciserver harbor]# 
[root@ciserver harbor]# tar -xzvf harbor-offline-installer-v1.10.4.tgz
harbor/harbor.v1.10.4.tar.gz
harbor/prepare 
harbor/LICENSE 
harbor/install.sh
harbor/common.sh
harbor/harbor.yml
[root@ciserver harbor]#
[root@ciserver harbor]# ls 
LICENSE  common common.sh docker-compose.yml harbor.v1.10.4.tar.gz harbor.yml install.sh prepare
[root@ciserver harbor]#

2. Configuration Harbor

1) harbor.yml

a. hostname: 192.168.56.100 
b. https 주석
#https: 
  #port: 443 
  #certificate: /your/certificate/path 
  #private_key: /your/private/key/path

- hostname - Harbor 접속 Domain

- https 주석 - https 사용하지 않을 경우 주석 disable 구성

2) common.sh

error "Need to install docker(17.06.0+) first and run this script again." 
#exit 1
error "Need to upgrade docker package to 17.06.0+." 
#exit 1

- 특정 OS에 맞는 특정 Docker version을 요구할 경우 위 exit 1을 주석처리 하여 기동 할 수 있다. 다만, 가능하면 요구하는 버전에 맞는 Docker 설치를 권고한다.

3. Harbor Install

(./install.sh --with-clair --with-chartmuseum)

[root@ciserver harbor]# ./install.sh --with-clair --with-chartmuseum

[Step 0]: checking if docker is installed ...
Need to upgrade docker package to 17.06.0+.

[Step 1]: checking docker-compose is installed ...

Note: docker-compose version: 1.25.5

[Step 2]: loading Harbor images ...
710700ba4a6b: Loading layer [==================================================>]  34.5 MB/34.5 MB
7fd57902f2bf: Loading layer [==================================================>] 8.465 MB/8.465 MB
9f7a3727b327: Loading layer [==================================================>]  67.5 MB/67.5 MB
b165ecbfa6a0: Loading layer [==================================================>] 3.072 kB/3.072 kB
618609e47ff5: Loading layer [==================================================>] 3.584 kB/3.584 kB
4941a988de67: Loading layer [==================================================>] 68.33 MB/68.33 MB
Loaded image: goharbor/chartmuseum-photon:v1.10.4
c249fd1745d2: Loading layer [==================================================>] 12.24 MB/12.24 MB
6f099dcc4dab: Loading layer [==================================================>] 42.51 MB/42.51 MB
eb32b6d20d4b: Loading layer [==================================================>] 5.632 kB/5.632 kB
5acd92618fef: Loading layer [==================================================>] 40.45 kB/40.45 kB
62b57401b9ca: Loading layer [==================================================>] 42.51 MB/42.51 MB
d7b6ded42cfb: Loading layer [==================================================>]  2.56 kB/2.56 kB
Loaded image: goharbor/harbor-core:v1.10.4
31b3ca7fa226: Loading layer [==================================================>]  63.6 MB/63.6 MB
b9972bab1402: Loading layer [==================================================>] 66.73 MB/66.73 MB
56b3ba4b4a66: Loading layer [==================================================>] 5.632 kB/5.632 kB
1654024d89fe: Loading layer [==================================================>]  2.56 kB/2.56 kB
040ec6bf5851: Loading layer [==================================================>]  2.56 kB/2.56 kB
e93cd0c30c28: Loading layer [==================================================>]  2.56 kB/2.56 kB
aed062c3be21: Loading layer [==================================================>]  2.56 kB/2.56 kB
820d1a1df842: Loading layer [==================================================>] 10.75 kB/10.75 kB
Loaded image: goharbor/harbor-db:v1.10.4
ce217f401320: Loading layer [==================================================>] 8.466 MB/8.466 MB
b324500c7da3: Loading layer [==================================================>] 3.584 kB/3.584 kB
042b5242fe78: Loading layer [==================================================>] 20.94 MB/20.94 MB
87dd45007ea3: Loading layer [==================================================>] 3.072 kB/3.072 kB
651d502d735c: Loading layer [==================================================>] 8.662 MB/8.662 MB
fe72a4614aa1: Loading layer [==================================================>] 30.42 MB/30.42 MB
Loaded image: goharbor/harbor-registryctl:v1.10.4
5de330f38841: Loading layer [==================================================>]  8.46 MB/8.46 MB
0af0ddd91395: Loading layer [==================================================>] 6.239 MB/6.239 MB
3685afd2d128: Loading layer [==================================================>] 16.04 MB/16.04 MB
d8057fcd0a39: Loading layer [==================================================>] 28.25 MB/28.25 MB
0340225731b6: Loading layer [==================================================>] 22.02 kB/22.02 kB
06d8d803f0eb: Loading layer [==================================================>] 50.52 MB/50.52 MB
Loaded image: goharbor/notary-server-photon:v1.10.4
76eab6dc7bf5: Loading layer [==================================================>] 332.6 MB/332.6 MB
c96d1ad1968e: Loading layer [==================================================>] 135.2 kB/135.2 kB
Loaded image: goharbor/harbor-migrator:v1.10.4
7426785037a5: Loading layer [==================================================>] 10.31 MB/10.31 MB
b9a0601e3558: Loading layer [==================================================>] 7.698 MB/7.698 MB
aac781885802: Loading layer [==================================================>] 223.2 kB/223.2 kB
8af4d736a2ab: Loading layer [==================================================>] 195.1 kB/195.1 kB
5fef45ce538d: Loading layer [==================================================>] 15.36 kB/15.36 kB
5f98131a71d5: Loading layer [==================================================>] 3.584 kB/3.584 kB
Loaded image: goharbor/harbor-portal:v1.10.4
528ae1964423: Loading layer [==================================================>] 12.24 MB/12.24 MB
b03ff000935f: Loading layer [==================================================>] 49.37 MB/49.37 MB
Loaded image: goharbor/harbor-jobservice:v1.10.4
6e2646825500: Loading layer [==================================================>] 89.65 MB/89.65 MB
fb20b8d71cf1: Loading layer [==================================================>] 3.072 kB/3.072 kB
d566c1cc124d: Loading layer [==================================================>]  59.9 kB/59.9 kB
c427dc7cb315: Loading layer [==================================================>] 61.95 kB/61.95 kB
Loaded image: goharbor/redis-photon:v1.10.4
6d6ba3b6ec7b: Loading layer [==================================================>] 85.27 MB/85.27 MB
7a5fdfe83ad0: Loading layer [==================================================>] 49.48 MB/49.48 MB
43de16c75891: Loading layer [==================================================>]  2.56 kB/2.56 kB
e27a79d7a642: Loading layer [==================================================>] 1.536 kB/1.536 kB
ab26083ef82a: Loading layer [==================================================>] 157.2 kB/157.2 kB
242de86f59b8: Loading layer [==================================================>] 3.017 MB/3.017 MB
Loaded image: goharbor/prepare:v1.10.4
9fd7cf078b16: Loading layer [==================================================>] 49.93 MB/49.93 MB
bffa9c13b070: Loading layer [==================================================>] 3.584 kB/3.584 kB
5bc5a2da3367: Loading layer [==================================================>] 3.072 kB/3.072 kB
d207162a345a: Loading layer [==================================================>]  2.56 kB/2.56 kB
3f5fa111d1ff: Loading layer [==================================================>] 3.072 kB/3.072 kB
6fac1f97e0a4: Loading layer [==================================================>] 3.584 kB/3.584 kB
39089450a8d3: Loading layer [==================================================>] 12.29 kB/12.29 kB
c43cc9ac71a3: Loading layer [==================================================>] 5.632 kB/5.632 kB
Loaded image: goharbor/harbor-log:v1.10.4
93dfe2d38dda: Loading layer [==================================================>] 115.3 MB/115.3 MB
a2d6890966ca: Loading layer [==================================================>] 12.15 MB/12.15 MB
008d8a39ac95: Loading layer [==================================================>] 3.072 kB/3.072 kB
a06e99290956: Loading layer [==================================================>] 49.15 kB/49.15 kB
6d0c609a7ea0: Loading layer [==================================================>] 3.584 kB/3.584 kB
cc7d9f19817b: Loading layer [==================================================>] 13.03 MB/13.03 MB
Loaded image: goharbor/clair-photon:v1.10.4
0c8c48462931: Loading layer [==================================================>] 8.466 MB/8.466 MB
7c096b7a5806: Loading layer [==================================================>]  9.71 MB/9.71 MB
f18d35335b53: Loading layer [==================================================>]  9.71 MB/9.71 MB
Loaded image: goharbor/clair-adapter-photon:v1.10.4
f55180240dc6: Loading layer [==================================================>] 10.31 MB/10.31 MB
Loaded image: goharbor/nginx-photon:v1.10.4
4a575c1c2167: Loading layer [==================================================>] 8.466 MB/8.466 MB
d0e9899aeeb5: Loading layer [==================================================>] 3.584 kB/3.584 kB
db6d9646f0e0: Loading layer [==================================================>] 3.072 kB/3.072 kB
478d5f29f1a6: Loading layer [==================================================>] 20.94 MB/20.94 MB
1fbbee6ba37e: Loading layer [==================================================>] 21.76 MB/21.76 MB
Loaded image: goharbor/registry-photon:v1.10.4
10bbb8d426b9: Loading layer [==================================================>] 14.61 MB/14.61 MB
91b66eb6b6b0: Loading layer [==================================================>] 28.25 MB/28.25 MB
58956c7bbf02: Loading layer [==================================================>] 22.02 kB/22.02 kB
1c86ba20384f: Loading layer [==================================================>] 49.09 MB/49.09 MB
Loaded image: goharbor/notary-signer-photon:v1.10.4


[Step 3]: preparing environment ...

[Step 4]: preparing harbor configs ...
prepare base dir is set to /root/harbor/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /secret/keys/secretkey
Generated certificate, key file: /secret/core/private_key.pem, cert file: /secret/registry/root.crt
Generated configuration file: /config/clair/postgres_env
Generated configuration file: /config/clair/config.yaml
Generated configuration file: /config/clair/clair_env
Generated configuration file: /config/clair-adapter/env
Generated configuration file: /config/chartserver/env
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir



[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating network "harbor_harbor-clair" with the default driver
Creating network "harbor_harbor-chartmuseum" with the default driver
Creating harbor-log ... done
Creating registryctl   ... done
Creating harbor-portal ... done
Creating chartmuseum   ... done
Creating harbor-db     ... done
Creating registry      ... done
Creating redis         ... done
Creating clair         ... done
Creating harbor-core   ... done
Creating nginx             ... done
Creating harbor-jobservice ... done
Creating clair-adapter     ... done
----Harbor has been installed and started successfully.----
[root@ciserver harbor]#

위와 같이 clair 도커 이미지 취약점 분석도구와 chartmuseum helm chart repository를 옵션으로 추가하여 함께 구성한다.

구성이 완료되면 다음과 같이 정상 기동되었는지 여부를 확인한다.

[root@ciserver harbor]# netstat -anp | grep 80 | grep LIST 
tcp6       0      0 :::80                   :::*                    LISTEN      4056/docker-proxy-c  
unix  2      [ ACC ]     STREAM     LISTENING     16940    805/NetworkManager   /var/run/NetworkManager/private-dhcp 
[root@ciserver harbor]# ps -efl | grep 4056 | grep -v grep 
4 S root      4056  1141  0  80   0 - 27245 futex_ 04:38 ?        00:00:00 /usr/libexec/docker/docker-proxy-current -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.19.0.10 -container-port 8080 
[root@ciserver harbor]#

port는 harbor.yml에 정의한 port 정보를 기반으로 확인하며, Process의 기동 상태를 점검한다.

이와 같이 기동된 Harbor는 기본 Docker Compose를 기반으로 기동된다. 다음과 같이 Docker Compose로 기동된 Process를 다시한번 확인해 보자.

[root@ciserver harbor]# docker-compose ps
      Name                     Command                  State                 Ports          
---------------------------------------------------------------------------------------------
chartmuseum         ./docker-entrypoint.sh           Up (healthy)   9999/tcp                 
clair               ./docker-entrypoint.sh           Up (healthy)   6060/tcp, 6061/tcp       
clair-adapter       /clair-adapter/clair-adapter     Up (healthy)   8080/tcp                 
harbor-core         /harbor/harbor_core              Up (healthy)                            
harbor-db           /docker-entrypoint.sh            Up (healthy)   5432/tcp                 
harbor-jobservice   /harbor/harbor_jobservice  ...   Up (healthy)                            
harbor-log          /bin/sh -c /usr/local/bin/ ...   Up (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       nginx -g daemon off;             Up (healthy)   8080/tcp                 
nginx               nginx -g daemon off;             Up (healthy)   0.0.0.0:80->8080/tcp     
redis               redis-server /etc/redis.conf     Up (healthy)   6379/tcp                 
registry            /home/harbor/entrypoint.sh       Up (healthy)   5000/tcp                 
registryctl         /home/harbor/start.sh            Up (healthy)                            
[root@ciserver harbor]#

Harbor를 구축하기 위해서는 위와 같이 12개의 Docker Process가 기동되어야 하며, 각각 구성하기 번거로운 점을 install.sh을 구성하여 손쉽게 구축할 수 있도록 Harbor는 제공하고 있다.

4. Harbor Log

특정 Harbor의 Log를 확인하고자 할 경우 /var/log/harbor 디렉토리를 확인한다.

[root@ciserver harbor]# ls -la /var/log/harbor/
total 128
drwxr-xr-x. 2 10000 10000   226 Aug  2 04:38 .
drwxr-xr-x. 9 root  root   4096 Aug  2 04:38 ..
-rw-r--r--. 1 10000 10000   224 Aug  2 04:38 chartmuseum.log
-rw-r--r--. 1 10000 10000   390 Aug  2 04:38 clair-adapter.log
-rw-r--r--. 1 10000 10000  5493 Aug  2 04:43 clair.log
-rw-r--r--. 1 10000 10000 14225 Aug  2 04:54 core.log
-rw-r--r--. 1 10000 10000  6686 Aug  2 04:38 jobservice.log
-rw-r--r--. 1 10000 10000 23809 Aug  2 04:54 portal.log
-rw-r--r--. 1 10000 10000  6028 Aug  2 04:38 postgresql.log
-rw-r--r--. 1 10000 10000  6913 Aug  2 04:54 proxy.log
-rw-r--r--. 1 10000 10000  4866 Aug  2 04:53 redis.log
-rw-r--r--. 1 10000 10000 18016 Aug  2 04:54 registry.log
-rw-r--r--. 1 10000 10000 15930 Aug  2 04:54 registryctl.log
[root@ciserver harbor]#

5. Harbor Dashboard

마지막으로 위와 같이 Harbor 대시보드를 확인한다. 정상적으로 기동될 경우 Harbor는 정상 설치된 것으로 볼 수 있다.

 Harbor Reconfig 

위와 같이 구성된 Harbor의 설정을 변경하고 싶을 경우 아래와 같이 적용이 가능하다.

예를 들어 Harbor port를 80에서 8080으로 변경하고 싶을 경우를 가정하고 살펴보자.

1) docker-compose down -v

[root@ciserver harbor]# docker-compose down -v     
Stopping harbor-jobservice ... done 
Stopping clair-adapter     ... done 
Stopping nginx             ... done 
Stopping harbor-core       ... done 
Stopping clair             ... done 
Stopping redis             ... done 
Stopping harbor-db         ... done 
Stopping registry          ... done 
Stopping chartmuseum       ... done 
Stopping harbor-portal     ... done 
Stopping registryctl       ... done 
Stopping harbor-log        ... done 
Removing harbor-jobservice ... done 
Removing clair-adapter     ... done 
Removing nginx             ... done 
Removing harbor-core       ... done 
Removing clair             ... done 
Removing redis             ... done 
Removing harbor-db         ... done 
Removing registry          ... done 
Removing chartmuseum       ... done 
Removing harbor-portal     ... done 
Removing registryctl       ... done 
Removing harbor-log        ... done 
Removing network harbor_harbor 
Removing network harbor_harbor-clair 
Removing network harbor_harbor-chartmuseum 
[root@ciserver harbor]#

먼저 Harbor를 기동하고 있는 docker-compose를 다운한다.

2) harbor.yml 파일 수정

다음으로 harbor.yml 파일을 재구성한 후 (port 80 → 8080) 저장한다.

3) prepare를 이용한 변경 사항 적용

[root@ciserver harbor]# ./prepare --with-clair --with-chartmuseum
prepare base dir is set to /root/harbor/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registry/root.crt
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Clearing the configuration file: /config/clair/postgresql-init.d/README.md
Clearing the configuration file: /config/clair/postgres_env
Clearing the configuration file: /config/clair/config.yaml
Clearing the configuration file: /config/clair/clair_env
Clearing the configuration file: /config/clair-adapter/env
Clearing the configuration file: /config/chartserver/env
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /secret/keys/secretkey
Copying offline data file for clair DB
Generated configuration file: /config/clair/postgres_env
Generated configuration file: /config/clair/config.yaml
Generated configuration file: /config/clair/clair_env
Generated configuration file: /config/clair-adapter/env
Generated configuration file: /config/chartserver/env
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
[root@ciserver harbor]#

prepare 적용 시 변경 된 구성을 적용할 수 있도록 reconfiguration이 동작한다.

4) docker-compose up -d

[root@ciserver harbor]# docker-compose up -d 
Creating network "harbor_harbor" with the default driver 
Creating network "harbor_harbor-clair" with the default driver 
Creating network "harbor_harbor-chartmuseum" with the default driver 
Creating harbor-log ... done 
Creating harbor-db     ... done 
Creating redis         ... done 
Creating harbor-portal ... done 
Creating registryctl   ... done 
Creating chartmuseum   ... done 
Creating registry      ... done 
Creating clair         ... done 
Creating harbor-core   ... done 
Creating nginx             ... done 
Creating harbor-jobservice ... done 
Creating clair-adapter     ... done 
[root@ciserver harbor]#

위와 같이 docker-compose up으로 Harbor를 기동한다.

5) 변경 사항 확인

위와 같이 8080으로 변경된 상태로 접속이 가능한 것을 확인할 수 있다.

다음 포스팅에서는 Harbor 활용 방안에 대해 살펴보도록 하자.