Kubernetes 설치 (Redhat8, Single Master)

1. Unique MAC 및 Product UUID 확인

- 각 서버들의 MAC과 UUID가 Unique해야 함

# ip link

or

# ifconfig -a

 

# sudo cat /sys/class/dmi/id/product_uuid

 

2. Swap 해제

# sudo swapoff -a

 

# vi /etc/fstab   // swap 부분을 삭제하거나 swap > none으로 변경

#UUID=a43e5836-d773-4ce9-81b7-c20003b6f506 swap                    swap    defaults        0 0

UUID=a43e5836-d773-4ce9-81b7-c20003b6f506 none                    swap    defaults        0 0

- none으로 변경 시 아래 같은 경고 발생

[WARNING Swap]: swap is enabled; production deployments should disable swap unless testing the NodeSwap feature gate of the kubelet

 

3. Network 설정

- /etc/hosts

192.168.8.41    kubemaster01

192.168.8.42    kubemaster02

192.168.8.51    kubeworker01

192.168.8.52    kubeworker02

192.168.8.53    kubeworker03

 

- iproute-tc 설치

# sudo dnf install -y iproute-tc

 

- iptables에서 Bridged Network을 확인할 수 있도록 설정

# cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

 

- overlay 및 bridge netfilter 적용

# sudo modprobe overlay
# sudo modprobe br_netfilter

 

- Bridged Network 관련 커널 파라미터 적용 및 reboot 후 영구 설정

# cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

 

# sudo sysctl --system

 

4. SELinux 해제

# sudo setenforce 0

# sudo sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config

# sestatus

 

5. CRI-O 설치 (Container Runtime Interface)

- Docker에서 docker-shim을 업데이트하고 있지 않아 K8S 1.20 버전 부터 docker 사용을 지양하고 있음

- Kubernetes가 기본 CRI로 CRI-O를 사용하므로 설치 진행

 

- cri-o 최신 버전 확인

https://github.com/cri-o/cri-o/releases

- CRI-O 버전과 설치할 OS 종류 파라미터 지정

# export VERSION=1.24  # cri-o 2022/06 현재 최신 버전

 

# export OS=CentOS_8

또는

# export OS=CentOS_7 / CentOS_8_Stream 등

 

# curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo

# curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo

 

# dnf info cri-o

# dnf install cri-o

 

- cri-o 서비스 가동

# sudo systemctl daemon-reload
# sudo systemctl enable crio
# sudo systemctl start crio

 

6. Kubernetes 패키지 설치 및 서비스 활성화

# cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch

#baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64

enabled=1
gpgcheck=1

repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

 

# dnf install kubelet kubeadm kubectl --disableexcludes=kubernetes

 

- kubectl 명령줄 자동 완성을 bash-completion 설정

# dnf install bash-completion

# kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null

 

7. 방화벽 설정

(1) Control Plane / Master Node

Protocol	Direction	Port Range	Purpose	Used By
TCP	Inbound	6443	Kubernetes API server	All
TCP	Inbound	2379-2380	etcd server client API	kube-apiserver, etcd
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	10259	kube-scheduler	Self
TCP	Inbound	10257	kube-controller-manager	Self

$ sudo firewall-cmd --permanent --add-port=6443/tcp

$ sudo firewall-cmd --permanent --add-port=2379-2380/tcp

$ sudo firewall-cmd --permanent --add-port=10250/tcp

$ sudo firewall-cmd --permanent --add-port=10251/tcp

$ sudo firewall-cmd --permanent --add-port=10252/tcp

$ sudo firewall-cmd --reload

 

(2) Worker Node

Protocol	Direction	Port Range	Purpose	Used By
TCP	Inbound	10250	Kubelet API	Self, Control plane
TCP	Inbound	30000-32767	NodePort Services	All

$ sudo firewall-cmd --permanent --add-port=10250/tcp

$ sudo firewall-cmd --permanent --add-port=30000-32767/tcp 

$ sudo firewall-cmd --reload

 

8. kubelet 서비스 가동

# sudo systemctl enable --now kubelet

# sudo systemctl start kubelet

> master 설정될 때까지 kubelet 서비스는 계속 가동 실패하고 재가동됨

 

9. Master 노드 설치

- Master 노드에서 수행

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

 

# kubeadm init --pod-network-cidr=172.16.0.0/16

[init] Using Kubernetes version: v1.24.1

[preflight] Running pre-flight checks

        [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly

[preflight] Pulling images required for setting up a Kubernetes cluster

[preflight] This might take a minute or two, depending on the speed of your internet connection

[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

[certs] Using certificateDir folder "/etc/kubernetes/pki"

[certs] Generating "ca" certificate and key

[certs] Generating "apiserver" certificate and key

[certs] apiserver serving cert is signed for DNS names [kubemaster01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.8.41]

[certs] Generating "apiserver-kubelet-client" certificate and key

[certs] Generating "front-proxy-ca" certificate and key

[certs] Generating "front-proxy-client" certificate and key

[certs] Generating "etcd/ca" certificate and key

[certs] Generating "etcd/server" certificate and key

[certs] etcd/server serving cert is signed for DNS names [kubemaster01 localhost] and IPs [192.168.8.41 127.0.0.1 ::1]

[certs] Generating "etcd/peer" certificate and key

[certs] etcd/peer serving cert is signed for DNS names [kubemaster01 localhost] and IPs [192.168.8.41 127.0.0.1 ::1]

[certs] Generating "etcd/healthcheck-client" certificate and key

[certs] Generating "apiserver-etcd-client" certificate and key

[certs] Generating "sa" key and public key

[kubeconfig] Using kubeconfig folder "/etc/kubernetes"

[kubeconfig] Writing "admin.conf" kubeconfig file

[kubeconfig] Writing "kubelet.conf" kubeconfig file

[kubeconfig] Writing "controller-manager.conf" kubeconfig file

[kubeconfig] Writing "scheduler.conf" kubeconfig file

[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

[kubelet-start] Starting the kubelet

[control-plane] Using manifest folder "/etc/kubernetes/manifests"

[control-plane] Creating static Pod manifest for "kube-apiserver"

[control-plane] Creating static Pod manifest for "kube-controller-manager"

[control-plane] Creating static Pod manifest for "kube-scheduler"

[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s

[apiclient] All control plane components are healthy after 13.002043 seconds

[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace

[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster

[upload-certs] Skipping phase. Please see --upload-certs

[mark-control-plane] Marking the node kubemaster01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]

[mark-control-plane] Marking the node kubemaster01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]

[bootstrap-token] Using token: jj8khu.4t7jk8z0netg25o9

[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles

[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes

[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials

[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token

[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster

[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace

[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key

[addons] Applied essential addon: CoreDNS

[addons] Applied essential addon: kube-proxy

 

Your Kubernetes control-plane has initialized successfully!

 

To start using your cluster, you need to run the following as a regular user:

 

  mkdir -p $HOME/.kube

  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

  sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

Alternatively, if you are the root user, you can run:

 

  export KUBECONFIG=/etc/kubernetes/admin.conf

 

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

  https://kubernetes.io/docs/concepts/cluster-administration/addons/

 

Then you can join any number of worker nodes by running the following on each as root:

 

kubeadm join 192.168.8.41:6443 --token jj8khu.4t7jk8z0netg25o9 \

--discovery-token-ca-cert-hash sha256:1da794df091b4d4428377e8418971b6cf084257afc9ab84accd22a333b094968

 

10. Master 노드 확인

# mkdir -p $HOME/.kube

# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

# sudo chown $(id -u):$(id -g) $HOME/.kube/config

# export KUBECONFIG=/etc/kubernetes/admin.conf

 

# kubectl get node
NAME           STATUS   ROLES           AGE   VERSION
kubemaster01   Ready    control-plane   5d    v1.24.1

 

11. Worker Node Join

(1) token을 저장하지 않았거나 token 만료 기간이 지났을 경우

- Master 노드에서 수행

- token 확인 후 필요 시 새로 생성

# kubeadm token list

# kubeadm token create --print-join-command

 

- discovery-token-ca-cert-hash 값이 없을 경우

# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'

 

(2) Master 노드 설치 후 출력된 명령어로 Worker 노드 추가

- Worker 노드에서 수행

kubeadm join 192.168.8.41:6443 --token jj8khu.4t7jk8z0netg25o9 \

--discovery-token-ca-cert-hash sha256:1da794df091b4d4428377e8418971b6cf084257afc9ab84accd22a333b094968

 

12. Worker 노드 확인

# kubectl get node
NAME           STATUS   ROLES           AGE   VERSION
kubemaster01   Ready    control-plane   5d    v1.24.1
kubeworker01   Ready    <none>          5d    v1.24.1
kubeworker02   Ready    <none>          5d    v1.24.1