Kubeflow를 활용하여 머신러닝 분석환경 구축하기

Kubeflow 란?

Kubeflow는 Kubernetes 환경에서 머신러닝을 분석할 수 있는 오픈소스 기계 학습 플랫폼이다. Kubeflow를 이용하면, 손쉽게 분석환경을 구축할 수 있으며, 직접 커스터마이징한 이미지를 구동하여 접속할 수 있는 대시보드 환경을 제공한다. kubeflow가 제공하는 다양한 기능에 대해서는 하나씩 구축해 나가면서 알아보기로 하자.

Minimum system requirements

먼저 Kubeflow 구축 과정이다. 앞서 이야기한데로 Kubeflow가 구성되기 위해서는 Kubernetes 환경이 사전에 구축되어 있어야 한다.

실제로 Kubeflow는 복잡한 시스템을 배포, 확장 및 관리하기 위해 Kubernetes를 기반환경으로 사용한다. Kubeflow는 Kubernetes 기반 서비스 제공하는 Public Cloud Managed Service와 On Prem, Local, Private Cloud 모든 환경에서 구성이 가능하다. 기본으로 다양한 Application 라이브러리와 스캐폴드 기술을 제공하며, 다양한 머신러닝 도구와 함께 동작할 수 있다.

a. Kubernetes 호환 버전

kubeflow 공식 홈페이지 : https://www.kubeflow.org/docs/started/k8s/overview/

현재 EKS가 제공하는 Kubernetes 버전은 1.17이며, 이는 Kubeflow 1.0 이상을 사용해야 호환 가능하다.

b. 권장 리소스 요구사항

Kubernetes Worker 노드의 권고 리소스 요구사항은 다음과 같다.

• 4 CPU
• 12GB 메모리
• 50GB 저장공간

현재 본 테스트에서 사용하는 EKS Worker Node는 c5.2xlarge로 배포되어 있으며, m5.large의 경우 CPU 부족으로 인해 Pod가 배치되지 않는 1 insufficient cpu, 1 insufficient pods와 같은 현상이 발생할 수 있으며, 추가적인 분석 도구 확장을 위해 c5.2xlarge 이상을 권고한다.

# GPU 버전을 사용하고 싶을 경우 P2/P3 등의 EC2 인스턴스를 사용해야 한다.

Kubeflow Install

kubeflow를 설치하기 위해서는 먼저 kubeflow cli 도구인 kfctl을 설치해야 한다. kfctl은 다음에서 다운로드 받을 수 있다.

https://github.com/kubeflow/kfctl/releases/tag/v1.2.0

다운로드 받은 kfctl을 다음과 같이 압축을 해제하고 /usr/local/bin에 복사한다.

[root@ip-192-168-114-198 kubeflow_install]# ls -la
total 30456
drwxr-xr-x  2 root root       50 Jan 19 07:47 .
dr-xr-x--- 17 root root     4096 Jan 19 07:46 ..
-rw-r--r--  1 root root 31181024 Jan 19 07:47 kfctl_v1.2.0-0-gbc038f9_linux.tar.gz
[root@ip-192-168-114-198 kubeflow_install]# tar -xzvf kfctl_v1.2.0-0-gbc038f9_linux.tar.gz 
./kfctl
[root@ip-192-168-114-198 kubeflow_install]# ./kfctl version
kfctl v1.2.0-0-gbc038f9
[root@ip-192-168-114-198 kubeflow_install]# cp kfctl /usr/local/bin/
[root@ip-192-168-114-198 kubeflow_install]# kfctl version
kfctl v1.2.0-0-gbc038f9
[root@ip-192-168-114-198 kubeflow_install]# 

다음으로 kubeflow를 설치하기 위한 env를 해당 console에 직접 등록하거나, profile에 등록한 후 적용한다.

export KF_NAME=KFTest
export BASE_DIR=/root/kubeflow
export KF_DIR=${BASE_DIR}/${KF_NAME}
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_k8s_istio.v1.1.0.yaml"

kfctl로 kubeflow를 설치하는 방법은 다음과 같다. 먼저 실행에 앞서 profile에 등록한 /root/kubeflow 하위에 KF_NAME으로 등록한 디렉토리를 생성한다. 이후 해당 경로로 이동하여 아래와 같은 명령어를 실행한다. 이때, 명령을 실행하는 디렉토리가 비어있어야 한다.

kfctl apply -V -f ${CONFIG_URI} : -V 옵션은 Verbose이며, -f 옵션은 CONFIG_URI에 등록한 yaml 파일을 적용한다.

[root@ip-192-168-114-198 KFTest]# kfctl apply -V -f ${CONFIG_URI}
INFO[0000] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_k8s_istio.v1.1.0.yaml to /tmp/767353531/tmp.yaml  filename="utils/k8utils.go:178"
INFO[0000] Downloading https://raw.githubusercontent.com/kubeflow/manifests/v1.1-branch/kfdef/kfctl_k8s_istio.v1.1.0.yaml to /tmp/120901854/tmp_app.yaml  filename="loaders/loaders.go:71"
INFO[0000] App directory /root/kubeflow/KFTest already exists  filename="coordinator/coordinator.go:270"
INFO[0000] Writing KfDef to kfctl_k8s_istio.v1.1.0.yaml  filename="coordinator/coordinator.go:273"
INFO[0000] No name specified in KfDef.Metadata.Name; defaulting to KFTest based on location of config file: /root/kubeflow/KFTest/kfctl_k8s_istio.v1.1.0.yaml.  filename="coordinator/coordinator.go:202"
INFO[0000] 
****************************************************************
Notice anonymous usage reporting enabled using spartakus
To disable it
If you have already deployed it run the following commands:
  cd $(pwd)
  kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus

For more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/
****************************************************************
  filename="coordinator/coordinator.go:120"
INFO[0000] Creating directory /root/kubeflow/KFTest/.cache  filename="kfconfig/types.go:450"
INFO[0000] Fetching https://github.com/kubeflow/manifests/archive/v1.1-branch.tar.gz to /root/kubeflow/KFTest/.cache/manifests  filename="kfconfig/types.go:498"
INFO[0002] Updating localPath to /root/kubeflow/KFTest/.cache/manifests/manifests-1.1-branch  filename="kfconfig/types.go:569"
INFO[0002] Fetch succeeded; LocalPath /root/kubeflow/KFTest/.cache/manifests/manifests-1.1-branch  filename="kfconfig/types.go:590"
INFO[0002] Processing application: namespaces            filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/namespaces  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: application           filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/application  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: istio-stack           filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/istio-stack  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: cluster-local-gateway  filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/cluster-local-gateway  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: istio                 filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/istio  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: cert-manager-crds     filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/cert-manager-crds  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: cert-manager-kube-system-resources  filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/cert-manager-kube-system-resources  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: cert-manager          filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/cert-manager  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: add-anonymous-user-filter  filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/add-anonymous-user-filter  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: metacontroller        filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/metacontroller  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: bootstrap             filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/bootstrap  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: spark-operator        filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/spark-operator  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: kubeflow-apps         filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/kubeflow-apps  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: knative               filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/knative  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: kfserving             filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/kfserving  filename="kustomize/kustomize.go:667"
INFO[0002] Processing application: spartakus             filename="kustomize/kustomize.go:569"
INFO[0002] Creating folder /root/kubeflow/KFTest/kustomize/spartakus  filename="kustomize/kustomize.go:667"
INFO[0002] /root/kubeflow/KFTest/.cache/manifests exists; not resyncing   filename="kfconfig/types.go:473"
INFO[0002] namespace: kubeflow                           filename="utils/k8utils.go:433"
INFO[0002] Creating namespace: kubeflow                  filename="utils/k8utils.go:438"
INFO[0002] Log cluster name into KfDef: NRSON-EKS-CLUSTER.ap-northeast-2.eksctl.io  filename="kustomize/kustomize.go:253"
INFO[0002] Deploying application namespaces              filename="kustomize/kustomize.go:266"
namespace/cert-manager created
Warning:  apply should be used on resource created by either  create --save-config or  apply
namespace/kubeflow configured
INFO[0003] Successfully applied application namespaces   filename="kustomize/kustomize.go:291"
INFO[0003] Deploying application application             filename="kustomize/kustomize.go:266"
serviceaccount/application-controller-service-account created
customresourcedefinition.apiextensions.k8s.io/applications.app.k8s.io created
clusterrole.rbac.authorization.k8s.io/application-controller-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/application-controller-cluster-role-binding created
service/application-controller-service created
statefulset.apps/application-controller-stateful-set created
application.app.k8s.io/application-controller-kubeflow created
INFO[0003] Successfully applied application application  filename="kustomize/kustomize.go:291"
INFO[0003] Deploying application istio-stack             filename="kustomize/kustomize.go:266"
namespace/istio-system created
configmap/istio created
configmap/istio-galley-configuration created
configmap/istio-security-custom-resources created
configmap/istio-sidecar-injector created
configmap/istio-install-parameters-5kmkd2f29g created
configmap/prometheus created
serviceaccount/istio-pilot-service-account created
serviceaccount/istio-security-post-install-account created
serviceaccount/istio-nodeagent-service-account created
serviceaccount/istio-multi created
serviceaccount/istio-mixer-service-account created
serviceaccount/istio-ingressgateway-service-account created
serviceaccount/prometheus created
serviceaccount/istio-citadel-service-account created
serviceaccount/istio-galley-service-account created
serviceaccount/istio-sidecar-injector-service-account created
customresourcedefinition.apiextensions.k8s.io/authorizationpolicies.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/policies.authentication.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotaspecbindings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotaspecs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/rbacconfigs.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/rules.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/serviceentries.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/meshpolicies.authentication.istio.io created
customresourcedefinition.apiextensions.k8s.io/serviceroles.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/sidecars.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/templates.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterrbacconfigs.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/httpapispecs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/httpapispecbindings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/handlers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/destinationrules.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/servicerolebindings.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/adapters.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/instances.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/attributemanifests.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created
clusterrole.rbac.authorization.k8s.io/istio-mixer-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-reader created
clusterrole.rbac.authorization.k8s.io/istio-citadel-istio-system created
clusterrole.rbac.authorization.k8s.io/prometheus-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-security-post-install-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-nodeagent-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-galley-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-sidecar-injector-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-pilot-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-mixer-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-nodeagent-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-galley-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-sidecar-injector-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-citadel-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-security-post-install-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-multi created
clusterrolebinding.rbac.authorization.k8s.io/istio-pilot-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-istio-system created
role.rbac.authorization.k8s.io/istio-ingressgateway-sds created
rolebinding.rbac.authorization.k8s.io/istio-ingressgateway-sds created
service/istio-pilot created
service/istio-citadel created
service/istio-galley created
service/istio-ingressgateway created
service/istio-policy created
service/istio-sidecar-injector created
service/istio-telemetry created
service/prometheus created
daemonset.apps/istio-nodeagent created
deployment.apps/prometheus created
deployment.apps/istio-galley created
deployment.apps/istio-ingressgateway created
deployment.apps/istio-pilot created
deployment.apps/istio-policy created
deployment.apps/istio-citadel created
deployment.apps/istio-telemetry created
deployment.apps/istio-sidecar-injector created
job.batch/istio-security-post-install-release-1.3-latest-daily created
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector created
destinationrule.networking.istio.io/istio-telemetry created
destinationrule.networking.istio.io/istio-policy created
horizontalpodautoscaler.autoscaling/istio-telemetry created
horizontalpodautoscaler.autoscaling/istio-ingressgateway created
horizontalpodautoscaler.autoscaling/istio-pilot created
horizontalpodautoscaler.autoscaling/istio-policy created
poddisruptionbudget.policy/istio-telemetry created
poddisruptionbudget.policy/istio-policy created
poddisruptionbudget.policy/istio-sidecar-injector created
poddisruptionbudget.policy/istio-ingressgateway created
poddisruptionbudget.policy/istio-galley created
poddisruptionbudget.policy/istio-pilot created
servicerole.rbac.istio.io/istio-ingressgateway created
servicerolebinding.rbac.istio.io/istio-ingressgateway created
attributemanifest.config.istio.io/kubernetes created
attributemanifest.config.istio.io/istioproxy created
handler.config.istio.io/kubernetesenv created
handler.config.istio.io/prometheus created
instance.config.istio.io/requestcount created
instance.config.istio.io/requestsize created
instance.config.istio.io/tcpbytereceived created
instance.config.istio.io/tcpconnectionsclosed created
instance.config.istio.io/requestduration created
instance.config.istio.io/responsesize created
instance.config.istio.io/attributes created
instance.config.istio.io/tcpconnectionsopened created
instance.config.istio.io/tcpbytesent created
rule.config.istio.io/promtcpconnectionopen created
rule.config.istio.io/tcpkubeattrgenrulerule created
rule.config.istio.io/promhttp created
rule.config.istio.io/kubeattrgenrulerule created
rule.config.istio.io/promtcpconnectionclosed created
rule.config.istio.io/promtcp created
INFO[0009] Successfully applied application istio-stack  filename="kustomize/kustomize.go:291"
INFO[0009] Deploying application cluster-local-gateway   filename="kustomize/kustomize.go:266"
namespace/istio-system configured
configmap/cluster-local-gateway-parameters-tbbdb2842d created
serviceaccount/cluster-local-gateway-service-account created
serviceaccount/istio-multi configured
clusterrole.rbac.authorization.k8s.io/istio-reader configured
clusterrolebinding.rbac.authorization.k8s.io/istio-multi configured
service/cluster-local-gateway created
deployment.apps/cluster-local-gateway created
horizontalpodautoscaler.autoscaling/cluster-local-gateway created
poddisruptionbudget.policy/cluster-local-gateway created
INFO[0010] Successfully applied application cluster-local-gateway  filename="kustomize/kustomize.go:291"
INFO[0010] Deploying application istio                   filename="kustomize/kustomize.go:266"
configmap/istio-parameters-cm9hckfgmb created
clusterrole.rbac.authorization.k8s.io/kubeflow-istio-admin created
clusterrole.rbac.authorization.k8s.io/kubeflow-istio-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-istio-view created
clusterrbacconfig.rbac.istio.io/default created
gateway.networking.istio.io/kubeflow-gateway created
serviceentry.networking.istio.io/google-api-entry created
serviceentry.networking.istio.io/google-storage-api-entry created
virtualservice.networking.istio.io/google-api-vs created
virtualservice.networking.istio.io/google-storage-api-vs created
virtualservice.networking.istio.io/grafana-vs created
INFO[0011] Successfully applied application istio        filename="kustomize/kustomize.go:291"
INFO[0011] Deploying application cert-manager-crds       filename="kustomize/kustomize.go:266"
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
INFO[0012] Successfully applied application cert-manager-crds  filename="kustomize/kustomize.go:291"
INFO[0012] Deploying application cert-manager-kube-system-resources  filename="kustomize/kustomize.go:266"
configmap/cert-manager-kube-params-parameters created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:webhook-authentication-reader created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
INFO[0012] Successfully applied application cert-manager-kube-system-resources  filename="kustomize/kustomize.go:291"
INFO[0012] Deploying application cert-manager            filename="kustomize/kustomize.go:266"
namespace/cert-manager configured
configmap/cert-manager-parameters created
serviceaccount/cert-manager created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:webhook-requester created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
apiservice.apiregistration.k8s.io/v1beta1.webhook.cert-manager.io created
application.app.k8s.io/cert-manager created
WARN[0014] Encountered error applying application cert-manager:  (kubeflow.error): Code 500 with message: Apply.Run : error when creating "/tmp/kout444800521": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server could not find the requested resource  filename="kustomize/kustomize.go:284"
WARN[0014] Will retry in 2 seconds.                      filename="kustomize/kustomize.go:285"
namespace/cert-manager unchanged
configmap/cert-manager-parameters unchanged
serviceaccount/cert-manager unchanged
serviceaccount/cert-manager-cainjector unchanged
serviceaccount/cert-manager-webhook unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:webhook-requester unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-view unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-edit unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:auth-delegator configured
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
service/cert-manager unchanged
service/cert-manager-webhook unchanged
deployment.apps/cert-manager unchanged
deployment.apps/cert-manager-cainjector configured
deployment.apps/cert-manager-webhook configured
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
apiservice.apiregistration.k8s.io/v1beta1.webhook.cert-manager.io unchanged
application.app.k8s.io/cert-manager configured
WARN[0017] Encountered error applying application cert-manager:  (kubeflow.error): Code 500 with message: Apply.Run : error when creating "/tmp/kout493229370": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request  filename="kustomize/kustomize.go:284"
WARN[0017] Will retry in 4 seconds.                      filename="kustomize/kustomize.go:285"
namespace/cert-manager unchanged
configmap/cert-manager-parameters unchanged
serviceaccount/cert-manager unchanged
serviceaccount/cert-manager-cainjector unchanged
serviceaccount/cert-manager-webhook unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:webhook-requester unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-view unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-edit unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:auth-delegator configured
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
service/cert-manager unchanged
service/cert-manager-webhook unchanged
deployment.apps/cert-manager unchanged
deployment.apps/cert-manager-cainjector configured
deployment.apps/cert-manager-webhook configured
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
apiservice.apiregistration.k8s.io/v1beta1.webhook.cert-manager.io unchanged
application.app.k8s.io/cert-manager configured
WARN[0021] Encountered error applying application cert-manager:  (kubeflow.error): Code 500 with message: Apply.Run : error when creating "/tmp/kout284290987": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request  filename="kustomize/kustomize.go:284"
WARN[0021] Will retry in 5 seconds.                      filename="kustomize/kustomize.go:285"
namespace/cert-manager unchanged
configmap/cert-manager-parameters unchanged
serviceaccount/cert-manager unchanged
serviceaccount/cert-manager-cainjector unchanged
serviceaccount/cert-manager-webhook unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:webhook-requester unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-view unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-edit unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:auth-delegator configured
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
service/cert-manager unchanged
service/cert-manager-webhook unchanged
deployment.apps/cert-manager unchanged
deployment.apps/cert-manager-cainjector configured
deployment.apps/cert-manager-webhook configured
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
apiservice.apiregistration.k8s.io/v1beta1.webhook.cert-manager.io unchanged
application.app.k8s.io/cert-manager configured
WARN[0027] Encountered error applying application cert-manager:  (kubeflow.error): Code 500 with message: Apply.Run : error when creating "/tmp/kout048781168": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request  filename="kustomize/kustomize.go:284"
WARN[0027] Will retry in 8 seconds.                      filename="kustomize/kustomize.go:285"
namespace/cert-manager unchanged
configmap/cert-manager-parameters unchanged
serviceaccount/cert-manager unchanged
serviceaccount/cert-manager-cainjector unchanged
serviceaccount/cert-manager-webhook unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:webhook-requester unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-view unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-edit unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:auth-delegator configured
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
service/cert-manager unchanged
service/cert-manager-webhook unchanged
deployment.apps/cert-manager unchanged
deployment.apps/cert-manager-cainjector configured
deployment.apps/cert-manager-webhook configured
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
apiservice.apiregistration.k8s.io/v1beta1.webhook.cert-manager.io unchanged
application.app.k8s.io/cert-manager configured
WARN[0035] Encountered error applying application cert-manager:  (kubeflow.error): Code 500 with message: Apply.Run : error when creating "/tmp/kout441774361": Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request  filename="kustomize/kustomize.go:284"
WARN[0035] Will retry in 18 seconds.                     filename="kustomize/kustomize.go:285"
namespace/cert-manager unchanged
configmap/cert-manager-parameters unchanged
serviceaccount/cert-manager unchanged
serviceaccount/cert-manager-cainjector unchanged
serviceaccount/cert-manager-webhook unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:webhook-requester unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-view unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-edit unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:auth-delegator configured
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
service/cert-manager unchanged
service/cert-manager-webhook unchanged
deployment.apps/cert-manager unchanged
deployment.apps/cert-manager-cainjector configured
deployment.apps/cert-manager-webhook configured
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
apiservice.apiregistration.k8s.io/v1beta1.webhook.cert-manager.io unchanged
application.app.k8s.io/cert-manager configured
clusterissuer.cert-manager.io/kubeflow-self-signing-issuer created
INFO[0053] Successfully applied application cert-manager  filename="kustomize/kustomize.go:291"
INFO[0053] Deploying application add-anonymous-user-filter  filename="kustomize/kustomize.go:266"
envoyfilter.networking.istio.io/add-user-filter created
INFO[0053] Successfully applied application add-anonymous-user-filter  filename="kustomize/kustomize.go:291"
INFO[0053] Deploying application metacontroller          filename="kustomize/kustomize.go:266"
serviceaccount/meta-controller-service created
customresourcedefinition.apiextensions.k8s.io/compositecontrollers.metacontroller.k8s.io created
customresourcedefinition.apiextensions.k8s.io/controllerrevisions.metacontroller.k8s.io created
customresourcedefinition.apiextensions.k8s.io/decoratorcontrollers.metacontroller.k8s.io created
clusterrolebinding.rbac.authorization.k8s.io/meta-controller-cluster-role-binding created
statefulset.apps/metacontroller created
INFO[0053] Successfully applied application metacontroller  filename="kustomize/kustomize.go:291"
INFO[0053] Deploying application bootstrap               filename="kustomize/kustomize.go:266"
configmap/admission-webhook-bootstrap-config-map created
serviceaccount/admission-webhook-bootstrap-service-account created
clusterrole.rbac.authorization.k8s.io/admission-webhook-bootstrap-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/admission-webhook-bootstrap-cluster-role-binding created
statefulset.apps/admission-webhook-bootstrap-stateful-set created
application.app.k8s.io/bootstrap created
INFO[0054] Successfully applied application bootstrap    filename="kustomize/kustomize.go:291"
INFO[0054] Deploying application spark-operator          filename="kustomize/kustomize.go:266"
serviceaccount/spark-operatoroperator-sa created
serviceaccount/spark-operatorspark created
customresourcedefinition.apiextensions.k8s.io/scheduledsparkapplications.sparkoperator.k8s.io created
customresourcedefinition.apiextensions.k8s.io/sparkapplications.sparkoperator.k8s.io created
clusterrole.rbac.authorization.k8s.io/spark-operatoroperator-cr created
clusterrolebinding.rbac.authorization.k8s.io/spark-operatorsparkoperator-crb created
role.rbac.authorization.k8s.io/spark-operatorspark-role created
rolebinding.rbac.authorization.k8s.io/spark-operatorspark-role-binding created
deployment.apps/spark-operatorsparkoperator created
application.app.k8s.io/spark-operator created
INFO[0057] Successfully applied application spark-operator  filename="kustomize/kustomize.go:291"
INFO[0057] Deploying application kubeflow-apps           filename="kustomize/kustomize.go:266"
2021/01/19 08:13:12 nil value at `valueFrom.configMapKeyRef.name` ignored in mutation attempt
2021/01/19 08:13:12 nil value at `valueFrom.secretKeyRef.name` ignored in mutation attempt
secret/mysql-secret-fd5gktm75t created
secret/mlpipeline-minio-artifact created
secret/metadata-db-secrets created
secret/katib-mysql-secrets created
secret/katib-controller created
configmap/pipeline-api-server-config-f4t72426kt created
configmap/metadata-db-parameters created
configmap/metadata-grpc-configmap created
configmap/metadata-ui-parameters created
configmap/ml-pipeline-ui-configmap created
configmap/mpi-operator-config created
configmap/notebook-controller-notebook-controller-config-h4d668t5tb created
configmap/kubeflow-pipelines-profile-controller-env-mgh6th2gff created
configmap/pipeline-install-config-2829cc67f8 created
configmap/seldon-config created
configmap/trial-template created
configmap/trial-template-labeled created
configmap/workflow-controller-configmap created
configmap/jupyter-web-app-jupyter-web-app-config-dhcbh64467 created
configmap/workflow-controller-parameters created
configmap/kubeflow-pipelines-profile-controller-code-gd97t2m5f5 created
configmap/kubeflow-config created
configmap/katib-config created
configmap/admission-webhook-admission-webhook-parameters created
configmap/pipeline-upstream-install-config-c958fm776d created
configmap/profiles-profiles-config-4mgcmtgk6t created
persistentvolumeclaim/metadata-mysql created
persistentvolumeclaim/mysql-pv-claim created
persistentvolumeclaim/minio-pvc created
persistentvolumeclaim/katib-mysql created
serviceaccount/kubeflow-pipelines-metadata-writer created
serviceaccount/ml-pipeline-scheduledworkflow created
serviceaccount/ml-pipeline-ui created
serviceaccount/ml-pipeline-viewer-crd-service-account created
serviceaccount/ml-pipeline-visualizationserver created
serviceaccount/mpi-operator created
serviceaccount/mxnet-operator created
serviceaccount/notebook-controller-service-account created
serviceaccount/pipeline-runner created
serviceaccount/profiles-controller-service-account created
serviceaccount/pytorch-operator created
serviceaccount/seldon-manager created
serviceaccount/tf-job-dashboard created
serviceaccount/tf-job-operator created
serviceaccount/ml-pipeline-persistenceagent created
serviceaccount/ml-pipeline created
serviceaccount/kubeflow-pipelines-cache-deployer-sa created
serviceaccount/kubeflow-pipelines-cache created
serviceaccount/katib-ui created
serviceaccount/katib-controller created
serviceaccount/jupyter-web-app-service-account created
serviceaccount/centraldashboard created
serviceaccount/argo created
serviceaccount/admission-webhook-service-account created
serviceaccount/kubeflow-pipelines-container-builder created
serviceaccount/kubeflow-pipelines-viewer created
serviceaccount/metadata-ui created
serviceaccount/argo-ui created
customresourcedefinition.apiextensions.k8s.io/tfjobs.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/mxjobs.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/notebooks.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/poddefaults.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/profiles.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/pytorchjobs.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/scheduledworkflows.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/seldondeployments.machinelearning.seldon.io created
customresourcedefinition.apiextensions.k8s.io/mpijobs.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/suggestions.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/trials.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/viewers.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/experiments.kubeflow.org created
clusterrole.rbac.authorization.k8s.io/katib-controller created
clusterrole.rbac.authorization.k8s.io/admission-webhook-cluster-role created
clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-view created
clusterrole.rbac.authorization.k8s.io/kubeflow-mpijobs-admin created
clusterrole.rbac.authorization.k8s.io/kubeflow-mpijobs-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-mpijobs-view created
clusterrole.rbac.authorization.k8s.io/kubeflow-mxjobs-admin created
clusterrole.rbac.authorization.k8s.io/kubeflow-mxjobs-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-mxjobs-view created
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-clusterrole created
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role created
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role created
clusterrole.rbac.authorization.k8s.io/kubeflow-pytorchjobs-admin created
clusterrole.rbac.authorization.k8s.io/kubeflow-pytorchjobs-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-pytorchjobs-view created
clusterrole.rbac.authorization.k8s.io/kubeflow-tfjobs-admin created
clusterrole.rbac.authorization.k8s.io/kubeflow-tfjobs-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-tfjobs-view created
clusterrole.rbac.authorization.k8s.io/kubeflow-katib-view created
clusterrole.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role created
clusterrole.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role created
clusterrole.rbac.authorization.k8s.io/ml-pipeline-ui created
clusterrole.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role created
clusterrole.rbac.authorization.k8s.io/mpi-operator created
clusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-admin created
clusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-edit created
clusterrole.rbac.authorization.k8s.io/notebook-controller-kubeflow-notebooks-view created
clusterrole.rbac.authorization.k8s.io/notebook-controller-role created
clusterrole.rbac.authorization.k8s.io/seldon-manager-role-kubeflow created
clusterrole.rbac.authorization.k8s.io/kubeflow-view created
clusterrole.rbac.authorization.k8s.io/argo created
clusterrole.rbac.authorization.k8s.io/argo-ui created
clusterrole.rbac.authorization.k8s.io/ml-pipeline created
clusterrole.rbac.authorization.k8s.io/mxnet-operator created
clusterrole.rbac.authorization.k8s.io/pytorch-operator created
clusterrole.rbac.authorization.k8s.io/tf-job-operator created
clusterrole.rbac.authorization.k8s.io/kubeflow-katib-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-katib-admin created
clusterrole.rbac.authorization.k8s.io/kubeflow-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-admin created
clusterrole.rbac.authorization.k8s.io/seldon-manager-sas-role-kubeflow created
clusterrole.rbac.authorization.k8s.io/katib-ui created
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-view created
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-edit created
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-kubeflow-notebook-ui-admin created
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-cluster-role created
clusterrole.rbac.authorization.k8s.io/centraldashboard created
clusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-view created
clusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-edit created
clusterrole.rbac.authorization.k8s.io/kubeflow-kubernetes-admin created
clusterrole.rbac.authorization.k8s.io/admission-webhook-kubeflow-poddefaults-admin created
clusterrolebinding.rbac.authorization.k8s.io/jupyter-web-app-cluster-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/katib-controller created
clusterrolebinding.rbac.authorization.k8s.io/katib-ui created
clusterrolebinding.rbac.authorization.k8s.io/centraldashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-clusterrolebinding created
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-ui created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding created
clusterrolebinding.rbac.authorization.k8s.io/admission-webhook-cluster-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/notebook-controller-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/profiles-cluster-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding created
clusterrolebinding.rbac.authorization.k8s.io/seldon-manager-sas-rolebinding-kubeflow created
clusterrolebinding.rbac.authorization.k8s.io/argo created
clusterrolebinding.rbac.authorization.k8s.io/argo-ui created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline created
clusterrolebinding.rbac.authorization.k8s.io/mxnet-operator created
clusterrolebinding.rbac.authorization.k8s.io/pytorch-operator created
clusterrolebinding.rbac.authorization.k8s.io/tf-job-operator created
clusterrolebinding.rbac.authorization.k8s.io/mpi-operator created
clusterrolebinding.rbac.authorization.k8s.io/seldon-manager-rolebinding-kubeflow created
role.rbac.authorization.k8s.io/metadata-ui created
role.rbac.authorization.k8s.io/kubeflow-pipelines-cache-role created
role.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-role created
role.rbac.authorization.k8s.io/ml-pipeline created
role.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-role created
role.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-role created
role.rbac.authorization.k8s.io/centraldashboard created
role.rbac.authorization.k8s.io/ml-pipeline-ui created
role.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role created
role.rbac.authorization.k8s.io/pipeline-runner created
role.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-role created
role.rbac.authorization.k8s.io/seldon-leader-election-role created
role.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role created
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-rolebinding created
rolebinding.rbac.authorization.k8s.io/centraldashboard created
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-binding created
rolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-metadata-writer-binding created
rolebinding.rbac.authorization.k8s.io/ml-pipeline created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent-binding created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-ui created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-binding created
rolebinding.rbac.authorization.k8s.io/pipeline-runner-binding created
rolebinding.rbac.authorization.k8s.io/seldon-leader-election-rolebinding created
rolebinding.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role-binding created
rolebinding.rbac.authorization.k8s.io/metadata-ui created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow-binding created
service/argo-ui created
service/centraldashboard created
service/jupyter-web-app-service created
service/katib-controller created
service/katib-db-manager created
service/katib-mysql created
service/katib-ui created
service/kubeflow-pipelines-profile-controller created
service/metadata-db created
service/metadata-envoy-service created
service/metadata-grpc-service created
service/metadata-service created
service/metadata-ui created
service/minio-service created
service/ml-pipeline created
service/cache-server created
service/ml-pipeline-visualizationserver created
service/admission-webhook-service created
service/notebook-controller-service created
service/profiles-kfam created
service/pytorch-operator created
service/seldon-webhook-service created
service/tf-job-operator created
service/mysql created
service/ml-pipeline-ui created
deployment.apps/cache-deployer-deployment created
deployment.apps/cache-server created
deployment.apps/centraldashboard created
deployment.apps/jupyter-web-app-deployment created
deployment.apps/katib-controller created
deployment.apps/katib-db-manager created
deployment.apps/katib-mysql created
deployment.apps/argo-ui created
deployment.apps/kubeflow-pipelines-profile-controller created
deployment.apps/metadata-db created
deployment.apps/metadata-deployment created
deployment.apps/metadata-envoy-deployment created
deployment.apps/admission-webhook-deployment created
deployment.apps/metadata-ui created
deployment.apps/metadata-writer created
deployment.apps/minio created
deployment.apps/ml-pipeline created
deployment.apps/ml-pipeline-persistenceagent created
deployment.apps/ml-pipeline-scheduledworkflow created
deployment.apps/ml-pipeline-ui created
deployment.apps/ml-pipeline-viewer-crd created
deployment.apps/katib-ui created
deployment.apps/mpi-operator created
deployment.apps/mxnet-operator created
deployment.apps/mysql created
deployment.apps/notebook-controller-deployment created
deployment.apps/profiles-deployment created
deployment.apps/pytorch-operator created
deployment.apps/seldon-controller-manager created
deployment.apps/tf-job-operator created
deployment.apps/workflow-controller created
deployment.apps/ml-pipeline-visualizationserver created
deployment.apps/metadata-grpc-deployment created
mutatingwebhookconfiguration.admissionregistration.k8s.io/seldon-mutating-webhook-configuration-kubeflow created
mutatingwebhookconfiguration.admissionregistration.k8s.io/admission-webhook-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/seldon-validating-webhook-configuration-kubeflow created
application.app.k8s.io/profiles-profiles created
application.app.k8s.io/minio created
application.app.k8s.io/mpi-operator created
application.app.k8s.io/mxnet-operator created
application.app.k8s.io/mysql created
application.app.k8s.io/centraldashboard created
application.app.k8s.io/kubeflow-pipelines created
application.app.k8s.io/pytorch-job-crds created
application.app.k8s.io/pytorch-operator created
application.app.k8s.io/katib-crds created
application.app.k8s.io/katib-controller created
application.app.k8s.io/tf-job-operator created
application.app.k8s.io/webhook created
application.app.k8s.io/argo created
application.app.k8s.io/notebook-controller-notebook-controller created
application.app.k8s.io/jupyter-web-app-jupyter-web-app created
application.app.k8s.io/tf-job-crds created
application.app.k8s.io/seldon-core-operator created
certificate.cert-manager.io/seldon-serving-cert created
certificate.cert-manager.io/admission-webhook-cert created
compositecontroller.metacontroller.k8s.io/kubeflow-pipelines-profile-controller created
destinationrule.networking.istio.io/ml-pipeline-ui created
destinationrule.networking.istio.io/ml-pipeline-visualizationserver created
destinationrule.networking.istio.io/ml-pipeline created
destinationrule.networking.istio.io/ml-pipeline-mysql created
issuer.cert-manager.io/seldon-selfsigned-issuer created
servicerole.rbac.istio.io/ml-pipeline-ui created
servicerole.rbac.istio.io/cache-server created
servicerole.rbac.istio.io/ml-pipeline-services created
servicerolebinding.rbac.istio.io/bind-gateway-ml-pipeline-ui created
servicerolebinding.rbac.istio.io/bind-ml-pipeline-internal created
servicerolebinding.rbac.istio.io/bind-cache-server-admission-webhook created
virtualservice.networking.istio.io/profiles-kfam created
virtualservice.networking.istio.io/ml-pipeline-ui created
virtualservice.networking.istio.io/jupyter-web-app-jupyter-web-app created
virtualservice.networking.istio.io/centraldashboard created
virtualservice.networking.istio.io/argo-ui created
virtualservice.networking.istio.io/metadata-ui created
virtualservice.networking.istio.io/katib-ui created
virtualservice.networking.istio.io/metadata-grpc created
INFO[0074] Successfully applied application kubeflow-apps  filename="kustomize/kustomize.go:291"
INFO[0074] Deploying application knative                 filename="kustomize/kustomize.go:266"
namespace/knative-serving created
secret/webhook-certs created
configmap/config-tracing created
configmap/config-domain created
configmap/config-gc created
configmap/config-istio created
configmap/config-logging created
configmap/config-defaults created
configmap/config-network created
configmap/config-observability created
configmap/config-autoscaler created
configmap/config-deployment created
serviceaccount/controller created
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev created
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev created
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev created
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev created
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev created
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev created
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev created
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev created
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev created
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev created
clusterrole.rbac.authorization.k8s.io/knative-serving-podspecable-binding created
clusterrole.rbac.authorization.k8s.io/custom-metrics-server-resources created
clusterrole.rbac.authorization.k8s.io/knative-serving-addressable-resolver created
clusterrole.rbac.authorization.k8s.io/knative-serving-admin created
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-view created
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-edit created
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-admin created
clusterrole.rbac.authorization.k8s.io/knative-serving-istio created
clusterrole.rbac.authorization.k8s.io/knative-serving-core created
clusterrolebinding.rbac.authorization.k8s.io/knative-serving-controller-admin created
clusterrolebinding.rbac.authorization.k8s.io/hpa-controller-custom-metrics created
clusterrolebinding.rbac.authorization.k8s.io/custom-metrics:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/custom-metrics-auth-reader created
service/controller created
service/activator-service created
service/autoscaler created
service/webhook created
deployment.apps/webhook created
deployment.apps/activator created
deployment.apps/autoscaler-hpa created
deployment.apps/controller created
deployment.apps/networking-istio created
deployment.apps/autoscaler created
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.serving.knative.dev created
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.serving.knative.dev created
validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.serving.knative.dev created
apiservice.apiregistration.k8s.io/v1beta1.custom.metrics.k8s.io created
application.app.k8s.io/knative-serving-crds created
application.app.k8s.io/knative-serving-install created
gateway.networking.istio.io/cluster-local-gateway created
horizontalpodautoscaler.autoscaling/activator created
image.caching.internal.knative.dev/queue-proxy created
servicerole.rbac.istio.io/istio-service-role created
servicerolebinding.rbac.istio.io/istio-service-role-binding created
INFO[0078] Successfully applied application knative      filename="kustomize/kustomize.go:291"
INFO[0078] Deploying application kfserving               filename="kustomize/kustomize.go:266"
secret/kfserving-webhook-server-secret created
configmap/inferenceservice-config created
configmap/kfserving-config created
customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kubeflow.org created
clusterrole.rbac.authorization.k8s.io/kfserving-proxy-role created
clusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-view created
clusterrole.rbac.authorization.k8s.io/manager-role created
clusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-admin created
clusterrole.rbac.authorization.k8s.io/kubeflow-kfserving-edit created
clusterrolebinding.rbac.authorization.k8s.io/kfserving-proxy-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/manager-rolebinding created
service/kfserving-controller-manager-service created
service/kfserving-webhook-server-service created
service/kfserving-controller-manager-metrics-service created
statefulset.apps/kfserving-controller-manager created
mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org created
validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kubeflow.org created
application.app.k8s.io/kfserving created
application.app.k8s.io/kfserving-crds created
certificate.cert-manager.io/serving-cert created
issuer.cert-manager.io/selfsigned-issuer created
INFO[0083] Successfully applied application kfserving    filename="kustomize/kustomize.go:291"
INFO[0083] Deploying application spartakus               filename="kustomize/kustomize.go:266"
configmap/spartakus-config created
serviceaccount/spartakus created
clusterrole.rbac.authorization.k8s.io/spartakus created
clusterrolebinding.rbac.authorization.k8s.io/spartakus created
deployment.apps/spartakus-volunteer created
application.app.k8s.io/spartakus created
INFO[0084] Successfully applied application spartakus    filename="kustomize/kustomize.go:291"
INFO[0084] Applied the configuration Successfully!       filename="cmd/apply.go:75"
[root@ip-192-168-114-198 KFTest]#

설치는 이것으로 완료이다. 정상적으로 Kubeflow가 구축이 되면 다음과 같이 Pod 상태를 확인할 수 있다.

[root@ip-192-168-114-198 kustomize]# kubectl get pod -A
NAMESPACE         NAME                                                         READY   STATUS      RESTARTS   AGE
cert-manager      cert-manager-cainjector-774bd85548-qzsrw                     1/1     Running     0          6m56s
cert-manager      cert-manager-f8f6c65f9-b79cj                                 1/1     Running     0          6m56s
cert-manager      cert-manager-webhook-59fb566ff-r5kdt                         1/1     Running     1          6m56s
istio-system      cluster-local-gateway-79fd6955d7-6j2t5                       1/1     Running     0          6m58s
istio-system      istio-citadel-6c468575db-c7929                               1/1     Running     0          7m2s
istio-system      istio-galley-68c944d6b4-nv87p                                1/1     Running     0          7m2s
istio-system      istio-ingressgateway-85d57dc8bc-9qhzd                        1/1     Running     0          7m2s
istio-system      istio-nodeagent-2hx77                                        1/1     Running     0          7m3s
istio-system      istio-nodeagent-wr692                                        1/1     Running     0          7m3s
istio-system      istio-pilot-77bc8867cf-ss6g4                                 2/2     Running     0          7m2s
istio-system      istio-policy-59d86bdf79-5zn2l                                2/2     Running     1          7m2s
istio-system      istio-security-post-install-release-1.3-latest-daily-tv7rl   0/1     Completed   0          7m1s
istio-system      istio-sidecar-injector-6746558cb8-w2hh5                      1/1     Running     0          7m1s
istio-system      istio-telemetry-6854d9f564-f5jqn                             2/2     Running     1          7m2s
istio-system      prometheus-6b55759ff5-5r9np                                  1/1     Running     0          7m3s
knative-serving   activator-559f5c99bd-2lb7b                                   1/1     Running     0          5m51s
knative-serving   autoscaler-7f5b5d65dc-gf7jn                                  1/1     Running     0          5m50s
knative-serving   autoscaler-hpa-5f5fb77d99-ntl9s                              1/1     Running     0          5m51s
knative-serving   controller-5bf7fcf7dc-79w6v                                  1/1     Running     0          5m50s
knative-serving   networking-istio-6dcd5c7f7f-bfm4b                            1/1     Running     0          5m50s
knative-serving   webhook-84bb88bf75-7zmqf                                     1/1     Running     0          5m51s
kube-system       aws-node-b88st                                               1/1     Running     0          40m
kube-system       aws-node-p6f7s                                               1/1     Running     0          40m
kube-system       coredns-7dd7f84d9-tr2zb                                      1/1     Running     0          54m
kube-system       coredns-7dd7f84d9-z5b6c                                      1/1     Running     0          54m
kube-system       kube-proxy-5t9lb                                             1/1     Running     0          40m
kube-system       kube-proxy-smm7f                                             1/1     Running     0          40m
kubeflow          admission-webhook-bootstrap-stateful-set-0                   1/1     Running     1          6m15s
kubeflow          admission-webhook-deployment-5bc5f97cfd-g5jm2                1/1     Running     0          5m56s
kubeflow          application-controller-stateful-set-0                        1/1     Running     0          7m6s
kubeflow          argo-ui-669bcd8bfc-pv8pm                                     1/1     Running     0          5m57s
kubeflow          cache-deployer-deployment-b75f5c5f6-6t7sn                    2/2     Running     1          5m57s
kubeflow          cache-server-85bccd99bd-kk8fx                                2/2     Running     0          5m57s
kubeflow          centraldashboard-68965b5d89-xjzlv                            1/1     Running     0          5m57s
kubeflow          jupyter-web-app-deployment-5dfbb68956-ltmcl                  1/1     Running     0          5m57s
kubeflow          katib-controller-76b78f5db-96xpk                             1/1     Running     1          5m57s
kubeflow          katib-db-manager-67c9554b6d-fdrpf                            1/1     Running     2          5m57s
kubeflow          katib-mysql-5754b5dd66-xwwng                                 1/1     Running     0          5m57s
kubeflow          katib-ui-844b4fc655-f8n6z                                    1/1     Running     0          5m54s
kubeflow          kfserving-controller-manager-0                               2/2     Running     0          5m46s
kubeflow          kubeflow-pipelines-profile-controller-65b65d97bb-9gr7v       1/1     Running     0          5m56s
kubeflow          metacontroller-0                                             1/1     Running     0          6m15s
kubeflow          metadata-db-695fb6f55-x97bp                                  1/1     Running     0          5m56s
kubeflow          metadata-deployment-7d77b884b6-ps96k                         1/1     Running     0          5m56s
kubeflow          metadata-envoy-deployment-c5985d64b-mw2cj                    1/1     Running     0          5m56s
kubeflow          metadata-grpc-deployment-9fdb476-vd8d8                       1/1     Running     3          5m52s
kubeflow          metadata-ui-cf67fdb48-j249h                                  1/1     Running     0          5m55s
kubeflow          metadata-writer-59d755696c-cxt4n                             2/2     Running     0          5m55s
kubeflow          minio-6647564c5c-wx6ck                                       1/1     Running     0          5m55s
kubeflow          ml-pipeline-6bc56cd86d-fj5kk                                 2/2     Running     0          5m55s
kubeflow          ml-pipeline-persistenceagent-6f99b56974-pqbtr                2/2     Running     0          5m55s
kubeflow          ml-pipeline-scheduledworkflow-d596b8bd-92l26                 2/2     Running     0          5m54s
kubeflow          ml-pipeline-ui-8695cc6b46-r75fb                              2/2     Running     0          5m54s
kubeflow          ml-pipeline-viewer-crd-5998ff7f56-hw4nv                      2/2     Running     2          5m54s
kubeflow          ml-pipeline-visualizationserver-cbbb5b5b-lm4gb               2/2     Running     0          5m52s
kubeflow          mpi-operator-c747f5bf6-vs4w9                                 1/1     Running     0          5m54s
kubeflow          mxnet-operator-7cd59d475-nglks                               1/1     Running     0          5m53s
kubeflow          mysql-76597cf5b5-ldmzs                                       2/2     Running     0          5m53s
kubeflow          notebook-controller-deployment-756587d86-m9g46               1/1     Running     0          5m53s
kubeflow          profiles-deployment-65fcc9c97-tdz4b                          2/2     Running     0          5m53s
kubeflow          pytorch-operator-5db58565b-sl985                             1/1     Running     0          5m53s
kubeflow          seldon-controller-manager-6ddf664d54-sqwrm                   1/1     Running     0          5m52s
kubeflow          spark-operatorsparkoperator-85bbf89886-5kdwv                 1/1     Running     0          6m12s
kubeflow          spartakus-volunteer-7566cfd658-p9pc6                         1/1     Running     0          5m45s
kubeflow          tf-job-operator-5bf84768bf-27q2k                             1/1     Running     0          5m52s
kubeflow          workflow-controller-54dccb7dc4-59n7h                         1/1     Running     0          5m52s
[root@ip-192-168-114-198 kustomize]#

Kubeflow는 총 4개의 Namespace를 생성한다. (istio-system, kubeflow, cert-manager, knative-serving)

위와 같이 한눈에 봐도 많아 보이는 Pod들이 기동되고, 그 중 istio-ingressgateway를 통해 kubeflow 대시보드에 접근할 수 있다.

[root@ip-192-168-114-198 KFTest]# kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}'
31380
[root@ip-192-168-114-198 KFTest]#

위와 같이 대시보드에 접근할 수 있는 NodePort를 확인할 수 있다. 물론 대시보드에 접근하기 위해 Service Type을 변경할 수 있으며, NodePort를 사용하더라도 직접 Worker Node에 접근이 가능하도록 보안그룹을 구성해야 한다.

Kubeflow Central Dashboard

Kubeflow 대시보드는 앞서 확인한데로 다음과 같이 접근이 가능하다.

http://[WORKER_NODE_IP]:[NODE_PORT]

Start Setup 버튼을 클릭하면

다음과 같이 Namespace를 지정할 수 있으며, Finash 버튼을 클릭하면 메인 화면으로 이동할 수 있다.

kubeflow를 통해 다양한 기능을 사용할 수 있지만, 그 중 앞서 JupyterHub에서 살펴보았던 Jupyter Notebook을 기동하는 방법에 대해 알아보도록 하자. 이전 JupyterHub 포스팅은 다음을 참고한다.

[Jupyter Package] Jupyter notebook 환경을 제공하는 Jupyterhub 활용 가이드

먼저 왼쪽 메뉴바를 선택한 후 Notebook Servers를 선택한다.

> + NEW SERVER

NOTEBOOK SERVER를 추가하기 위해 아래와 같은 설정을 구성한다.

• Name : Notebook Server Name과 Namespace 지정
• Image : Jupyter Nodebook 이미지 지정 (기본으로 Tensorflow 라이브러리가 포함된 4개의 버전이 포함되어 있으며, 추가로 커스텀 이미지를 등록하여 기동할 수 있다.)
• CPU/RAM : 이미지에 할당할 CPU/MEMORY를 지정
• Workspace Volume : Persistent Volume 영역을 지정 (노트북을 종료할 경우 데이터를 삭제하길 원한다면, Don`y user를 체크한다.)
• Data Volumes : 추가로 할당하기 원하는 볼륨 영역을 지정
• GPUs : GPU 사용 여부 지정 (필요시 GPU 수를 지정하며, 이를 사용하기 위해서는 AWS에서는 EC2 P2/P3 등의 Type을 선택하여 Worker Node를 구성해야만 가능하다.)
• 마지막으로 Shared Memory 사용 여부를 선택하고 LAUNCH 선택

위와 같이 추가한 Jupyter Nodebook이 기동되는 것을 볼 수 있다. 약 1분 정도 시간이 지나면 Notebook Server가 기동된 것을 확인할 수 있다.

위와 같이 기동이 완료되면 CONNECT 버튼으로 접속이 가능하다. 접속 전 Pod로 기동된 Jupyter Nodebook을 확인해 보자.

[root@ip-192-168-114-198 KFTest]# kubectl get pods -n kfsrv
NAME                                               READY   STATUS    RESTARTS   AGE
jupyternodebook-kubeflow-test-0                    2/2     Running   0          112s
ml-pipeline-ui-artifact-bd978746-dxpfr             2/2     Running   0          30m
ml-pipeline-visualizationserver-865c7865bc-5ql5h   2/2     Running   0          30m
[root@ip-192-168-114-198 KFTest]#

위와 같이 jupyternodebook-kubeflow-test-0이 정상 기동된 것을 확인할 수 있다. CONNECT 버튼을 클릭하면 다음과 같이 Jupyter Notebook 대시보드를 확인할 수 있다.

이후 대시보드를 통해 아래와 같이 활용할 수 있다.

Kubeflow는 JupyterHub와 같이 분석도구를 통합으로 관리할 수 있다.

그밖에 다양한 기능을 통해 머신러닝 분석 도구를 제공하는 유용한 툴로써 확장성을 갖고 활용될 수 있을 것이다.