Kubernetes - Multi-node cluster setup on AWS - Day 13

 
Kubernetes - multi-node cluster set up - 2-02-2021

Managed services providers
--------------------------
Setup
multi-node cluster
- CNI (Networking)
- Overlay
- CoreDNS


Cloud providers - AWS, AZURE, GCP

minikube -> deployment -> POD deploy -> Expose


AWS -> EKS -> Click -> Entire setup done

launch the cluster on aws cloud
- EBS (storage class)


Things fail, hard to fix, troubleshoot...
- keep practicing ..
- fail ..
- error come up ..
- analyze why you got error
- learn the concept
- implement the finding
- lesson learned..
- document the error and solution



k8s ->
is a container management tool





c1  | c2  | c3 | ...
----------------
Container Engine/
Container runtime  -> Docker/containerd/CRI-O/Rocket
---------
OS  -> Worker Node
----------------
physical server (Ram/CPU)

For user perspective, it does not matter what engine you use.
----------------
- kubectl is the command for any container engine..



Container Runtime interface
---------------------------

Node1     Node2    Node3
-----    -----    -----

When you have one container goes down, RS/RC will relaunch
- What is entire ndoes goes down, it will launch container on different node.

kube-controller
Controller Manager
-------------------
- Keep checking the health of worker node
 
- We will install one program on worker node. - keep checking are you alive. Keep on checking the state- are you alive
  - This program is kubelet.

Kubelet
- keep on replying master node (controller manager) that I am alive.

Lets say user want to launch 3 pods ->
  - user need to talk to API-Server on master node
  - API-server take a request and schedule by going to controller manager (scheduler) and


Whole set up is a cluster.

We will set up three nodes
- k8s master
- Worker 1
- Worket 2

- Worker node provide resources and also work is perform at worker node.
-

Master Node has following services
- API
- CM
- Scheduler
- ETCD


- User sents request to k8s master API-server.
- Based on rule set up, scheduler will schedule where to launch the POD.

On worker node, these service runs
kubelet (agent)
- acept the request from master node
- reports to kube-api on master node
- Checks the health of nodes

also work on worker node
networking concept
- kube-proxy
- overlay


For local installation
- Install 3 linux OS


Implement on AWS
- Login to AWS console

Plan: Launch 3 instances
  1 - master node
  2 - worker nodes

- First launch a node in Northern Virginia (or any region)
- select aws -> t2 micro 1 cpu/1GB  (recommended 2GB RAM/2 CPU) - free tier
- select 3 instances

name: kubernetes
enable all firewall for this lab

or enable one by one respective ports
- review and launch
create a provate key and save.

now, you have three instances

just pick one server

- you have to pick what container engine you want to use.
  - We will use container enginer - docker but you can use CRI-O or any

On master node, you will have api, cm, scheduler, etcd
- you can run apr, cm, schedurer, etcd on different container
- thats why master also needed container engine.

so we need to install container engine on all master and worker nodes.

rename your nodes
one kube-master
two kube1/2

get the public address and login using putty
use private key to login. (or use mobaX)

# yum install docker -y
aws has already yum repo setup

# systemctl enable docker --now
# yum install docker-ce -y  -> if its local
# docker info

command shows the output.

set up cluster using kubeadm command
- you can set up multinode cluster

# yum install kubeadm
it says not available.
# yum repolist

its not available, lets set up kubenetes with kubeadm

copy the kuberntes.repo repo code from kubernetes.io site and create a repo

# yum clean all
# yum repolist

now, you have more software pkgs are available
# yum install kubeadm

if still not available
--disableexclide

# yum install kubeadm

time to set up cluster now,

- we have master to manage worker node
- worker nodes to deploy containers
- master node can also be use to launch containers.
   (api,etcd,scheduler run on top of master)
- container management is done by kubelet.
  kubelet need to run on all the nodes. (master or worker)
# rpm -q kubelet
# systemctl status kubelet
# systemctl enable kubelet --now
# systemctl status kubelet
you will see activing status ..


Master node have lots of program running and they are running on their own containers.

now, we need to download images
# docker images

helps use to downoad image from master or
# kubeadm config images pull
# docker images
now, you see lots of images running.

# docker ps

now, set up the cluster - initialize the cluster. Run at master
# kubeadm init

it will set up the master.

it is an umbrell of multiple services.

You see it failed and none of the service starts.

lets review error one by one.

@minikube
> kc get pods


when pod runs, you will see IP is assigned.
- who decides this IP range?
- who decides network or CIDR?
  172.16.0.0/16
           .2
           .3

Basically it was master node who decides the network range.
It allocates the IP address for POD
 range of IP address POD gets.


# kubeadm init --help

# kubeadm init --pod-network-cidr=10.240.0.0/16

failed again..

error
detected "cgroupfs" as the docker cgroup driver...

kubernetes does not support docker but systemd


# docker info
cgroup driver: cgroupfs
( using cgroup - we can control the perofrmance of the process)

for container run time interface, we have to tell not to use crio


Go to docker config file
# cd /etc/docker
# vi daemon.json

google - how to change cgorup drive docker to systemd
{
  "exec-opts" : ["native.cgroupdriver=systemd"]
}

restart the docker service
# systemctl restart docker
# docker info | grep driver

now, it is now systemd

one of the pre-requisites is done.

now run kube init command again
# kubeadm init --pod-network-cidr=10.240.0.0/16

error tc not found in system path...

for this we have to install one of the software - tproute-tc

# yum install iproute-tc

# iproute
# tc

you get output
# kubeadm init --pod-network-cidr=10.240.0.0/16
got error again,
complains, you need 2 cpus

there is an option --ignore-preflight-errors
just skip the error if you get this error.

so, we are ignoring memory error.

# kubecadm init --pod-network-cidr=10.240.0.0/1 --ignore-preflight-errors=NumCPU --ignore-preflight-error=Mem
......



you will get join link. keep it..


finally we just set up master node.


Test it
# kubectl get pods


it failed. kubectl need to know ip of master, port .. credentials as well.

to conifgure, you have to copy admin file on your home directory under .kube/config


# mkdir -p #HOME/.kube
cp -o /etc/kubetneres/admin.conf $HOME/.kube.config
chown $HOME


# kc get pods
you see lots of containers running...

kube-api, kube-system
kube-controller, kube-cheduler and more ..


# kc get nodes

it show one value returned, and it is master node.
and if you see the status: not ready
you have to enable some networking services. view the output when you ran the kubeadmin command.


now go to worker node 1
get the ip and ssh to it.

# yum install docker -y
# yum enable docker -online
systemctl enable kubelet

# kubeadm join

who is the master?

# kc get
get some security info from master to client node to join...
send request with my security code to join...

the security code is also called tocken.

go to master node, run
# kubeadm tocker list

get the output and review
kubeadm init also get you tocket along with ip of master node..

if you foggot keep
# kubeadm tocket create --print-join-command
it will give you new tocket to join

use this output to join the master node. run it on worker node.

we see new errors

when you run this command, it does pre-glight check.. checks the error

we have to set up systemd driver

# vi /etc/docker/daemon.json

add the driver inf here

restart docker service
# systemctl restart docker

join again
- again error about tc
# yum install iproute-tc

networking bridging is disable, you have to enable

# sysctl -a | grep
kernel setting about bridging..

vi /etc/sysctl.d/k8s.conf
net.bridge.....

run join command again,
# kubeadm join ip ==tocken ...

finally get message
your node join the cluster...

there is no error as of now.

Go and confirm on master

# kc get nodes

you get new node added.

roles - none mean worker node.


status is still NotReady

On master, run
# kc get pods
 no pod running
# kc create deploy myd --image=httpd
# kc get deploy
deployment created
# kc get pods
pod on pending status

# kc descriribe pod myd.....

your worker node is not yet ready

you see the error under events:

failingscheduling..

on master node, scheduler is looking for node to deploy.

taint and tolerance...

this erro show that worker node is not available...

# kc get nodes



Now, you have to set up or add on cni - flannel

you ave to run url to run...

it will set up flannel

# kc apply -f https://raw/githubusercontent.com/coreos/flannel/master/documentaion/kube-flannel.yaml

# kc get nodes
# kc get pods
# kc get pods -o wide

# kc get svc
# kc expose deploy myd --port=80 --type=NodePort

for outside connection

anyone who knows the ip and port, can connect


since this node is on aws, we have public ip


now, we have to configure another worker node same way.