Kubectl day 43 - CRI-O, etcd HPA

 Multi-Node

today's topic CRI-O, etcd/HPA

aws -> ubuntu instance

t2 medium 2/4

instances - 3 -> 1 master, 2 worker nodes

tag
name: ubuntu_kube_CRI-O


attach/create private key

How to configure multi-node cluster?

we have master node and attach nodes
- by default we will have docker engine and will replace with CRI-O




On your PC, launch your minikube
> minikube start



Get the document, and fllow

copy the IP of your cloud instance, and login

- configure repo (apt)
 - cri-o -software setup

$ apt update
updates the repo

$ apt install -qq -y cri-o crio-runc cri-rools

go to cri-o
you get software for different OS flavor


# systemctl daemon-repload
# systemctl enable --now crip
# systemtl status crio    # our container enginer is started

instead of docker, we are using crio here.

almost the commands are same

type cri and tab

# crictl images
# crictl ps
# crictl pull httpd (pulls image)
# crictl images

technicaly, commands are very same, if you know docker, you can use

we don't use CRI-O directly but we ask k8s to manage it,.


setup another repo for k8s where kubernetes software is
for kubelet, kubeadm

start master
# kubeadm init --apiserver .....-advertise-address=<master_IP> --pod-network-cidr=192.168.0.0/16

you may  have to do some ip forwarding stuffs

setup some driver to use (overlay) - follow document



restart crio service
systemctl daemon-reload
systemctl enable --now crio

restart firewall
systemctl disable --now ufw

now, rerun kubeadm init --api....

your master is ready

keep the output safe
2. Now, go to another cloud system (worker node1)
login

you can put all commands together in shell and run or convert into play on ansible (playbook) and execute


set up repo as before for cri-o software to be downloaded

apt-install -qq -y cri-o cri-o-runc

systemctl daemon-reload
systecmtl enable --now crio

setup repo for kube software as well.

modeprobe overlay

swap off and firewall disable

kubeadm init ......


@master
kc get pods
# kc get nodes

you see only one master node
# kc describe nodes ip-172....

look for runtime container: CRI-O

you need overlay network (flannel)

we will use calico today

follow the doc ....


get the join command from master and go to worker node and paste


@worker node
kubeadm join <ip:port> --tocken .....

go to @master and run
# kc get nodes
you see master and worker nodes


# ps -aux | grep kubelet
you will see output for container run-time engine as CRI-O


# crictl ps

you see lots of containers running
# crioctl images # you see lots of images

# kc create deplopymnet myd --image=httpd
# kc get pods

everything is same but the contianer is managed by cri-o.

# kc get pods -o wide
# kc get


go to node

# crictl ps
# crictl images

# crictl exec ....
# crictl exec -lt <pod> sh


==============================

etcd/HPA

Service Mesh

kube
 - pod
 - secret
  - quota/limit

everything is stored in database
name of the database is etcd

this database is stored inside pod

> kc get pods -n kube-system

you see etcd pod

etcd is third party tool.. you can install on your own too.
- its a simple database with key=value format

everything is stored in etcd

if you plan to migrate or upgrade, you first backup the etcd and upgrade.
once upgrade is done, you restore it.

This is one of the way.
How to create backup/restore

# kc get pods
# kc get pods -o wide
# kc get nodes


Lets go to etcd

> kc -n kube-system exec -it etcd-minikube -- sh

you are inside etcd pod

# etcdctl
you will see help

kubernetes creates etcd with some security

describe the pod

> kc describe pod kube-system ...


go under etcd
you see listen-clietn-urls ..

you will also see the certificates

you need private key
server cert
and ca.crt

/var/lib/minukube/

go inside the pod
> kc -n kube-system exec -lt etcd-minukube -- sh

# cd /var/lib
there is no ls command

put some data on database

# etcdctl put xyz Sam

(it will add this data to database)

it will fail, because this database is secure, need username/pw.

follow etcd and hpa document.

<.crt" --key="


take a backup
snapshot save </var/tmp/dtcd.bk.db>


snapshot status
# etcdctl snapshot status
you have to specify cert, key as wel


to restore
# etcdctl <key-info> snapshot restore /var/tmp/etcd.bk.db --data-dir=/var/tmp/etcd.db

it will restore to /var/tmp/etcd.db

kubernetes is a tool to manage pods, database and more..

for migration and upgrade, you make a backup of etcd database

-------------------------------------
HPA -
---

scalling ->
we have one pod running
- load increases, we add new pod with replica SEt managed by deployment. it will scale out
- no load - scale down.

we have to keep on monitoring manually because we want if some traffic comes, we have enough capacity to handle the load.

we don't have metric server thats why we have to keep checking


> kc get pods -n kube-system

you will see 'metric server '


> kc top pods

our metric server, keep on monitoring our resources.

in kubernetes programs are called respurces

keep on checking pod, if cpu is more than 90% cpu load, (metric server monitors), something triggers to create a new pod.

if cpu goes over 90% keep on launching a new pod.

but note: You have to limit the max limit otherwise you may encounter problem. say 10 max.



we have one of the respurces HPA to do this job

clean your environment.
# kc get hpa
# kc delete hpa --all
# kc delete all --all

( Google for horizontal pod autoscale walkthough - get link at kubernetes.io)
auto-scaling

follow the doc

> notepad h1.yaml

> kc create -f h1.yaml

> kc get pods

create horizontal pod scaler

> kubectl autoscal deployment pho-apache --cpu-percent=50 --min=1 --max=10

hpa has been comfigured
> kc get hpa

target is unknown

you need to have metric server already configured.
> kc top pods

your pod has 0 utilization
when load increase, it will launch a new pod

to increase the load,
go to section increase load on doc

> kc get pods
> kc get hpa
you see new pod

> kc top pods

> kc get hpa
look the value under targets

stop the load and gradually pod will start disapearing ..

============
review

RBAC -> Rolebinding

high availibity cluster

node affinity rule

================================


etcd/HPA

ssh-key generate with passphrase

Check to see if ssh-key has passphrase is set up

New security implementaton force us to check if anyone is using their keys without passphrase. It was a big blow to our team. Some developers using windows as a base OS and some using linux as base os. On top of that the putty server session is saved using the key for all servers. so, for windows user, I had to generate key on their PC using puttygen and upload their public key to linux server and convert it to linux format. Finally append it to authorized_keys file.


check
$ sudo ssh-keygen -y -f /path/to/file

$ sudo find /usr/people/*/.ssh \( -name id\*a -a -o -name id\9 \) -exec echo "Checking: {}" \; -exec ssh-keygen -v -y -f {} \;

$ sudo find ~jay/.ssh \( -name id\*a -o -name id\*9 \) -exec ssh-keygen -P "" -u -f {} \;
$ find ./*/.ssh \( -name id\*a -o -name id\9) -exec ssh-keygen -y -f {} \;

Generate public and private key with passcode
$ sudo ssh-keygen -n [passphrase]

keep entering passphrase is not easy. Its a painful.
$ evel $(ssh-agent)
$ ssh-add ~sam/.ssh/id_rsa

Oracle Patch - How to copy

 
Transfer oracle patch and extract
1. Download the patch from oracle.com
2. Ftp patch to your repo server (/tmp/patches)
3. Unzip the patch
   # for z in p*.zip; do unzip -tq $z; done
4. Copy over to repo location
   # mv -v p*.zip /repo/software/oracle/Linux/x86_64/

geet - Super hit Nepali songs

 Lyrics:
जाऊ उमेर बेची सपना किनेर ल्याऊ
यी आँसुका कलशको साइत गरेर जाऊ
म हुँ प्रेमधागो चङ्गा भएर जाऊ
हिमताल हुँ म तिम्रै
तिमी गङ्गा भएर जाऊ

छोटो छ जिन्दगी फोटो खिचेर राख
छिट्टै फर्की आउँछु म, मुटु थिचेर राख
छोटो छ जिन्दगी फोटो खिचेर राख

अन्तरा १
धनको लिएर आशा मनको गर्यौ तमासा
म मुनासरी रुँदै छु बुझ आँसुको यो भाषा
परदेशको हुरीले निभ्ने हो कि प्राण सित्तै
बरु फर्की आउ रित्तै मधुमास यो नबित्दै
सय लाखको यो जुनी कौडीमा नफ्याँक

छिट्टै फर्की आउँछु म, मुटु थिचेर राख
छोटो छ जिन्दगी फोटो खिचेर राख

अन्तरा २
तिर्खाहरु सँगाली पानी बनाउदै छु
आफै जलेर हीराखानी बनाउँदै छु
रित्तो छ आज बादल फर्किन्छ क्यै लिएर
आँगन उज्यालिनेछ मुस्कान फैलिएर

तबसम्म दिलको चाह दिलमै मिचेर राख
छिट्टै फर्की आउँछु म, मुटु थिचेर राख
छोटो छ जिन्दगी फोटो खिचेर राख....

 --------------------------- 

फुलबुट्टे सारी
टाढा भएपनि
माया मारेर
के माया लाग्छ र ?
ओए काजल
ए है माया
नबोली नबोली
सोचे झै जिन्दगी रैनछ
उनको प्रीति
देखेर तिमीलाई

फुटेका चुरा
बदला बरीलै

पछ्यौरी लिसियो
झ्यालमा पर्दा छ
चौकादाउ
माया गरौंला
मिनिरल वाटर
छोटो छ जिन्दगी
म हाँसे हास्छेउ र ?
रेलको बाटो

=======================

Maanchhe kaa gunaasaa

हेरेर खुसी लाग्यो,केहि मज़्ज़ा आयो, केहि शैली सिक्न पनि पाइयो|

यस्ता यथार्थ सोच र क्षमता भएको कलाकार ले पाउनुपर्ने स्थान पायको छ।

सुगन्ध मिसिएन भन्दैमा सुन ले आफ्नु गुण छोड्दैन ।। वा वा

-------------
कस्तो हुदोरहेछ मान्छे को मन पनि  लामो समय पछि यो जोडी लाई  देख्दा खेरि आँखा बाट आसु नै आयो  यो जोडी लाई  यसरिनै रेर पाउ
एति राम्रो जोडि एति सुहाको जोडि किन छुटेको थ्यो होला बिचरा जे होश ऐले फेरि सङगै देखिन्दै रैछ सधै को लागि हो कि एक्छिन को लागि हो भर्सक सङगै मिलेर सधै भरी को लागि बसे धेरै राम्रो हुन्थ्यो होला सबै को लागि
मायाको भाब झल्किन्छ हामीले हेर्दा पुरानो हुदैन माया सधै ताजा रहोस् शुभकामना
एक अर्का लाई   आदर सम्मान  गरेको देख्दा  म

तिम्रो मायाँ सन्धी भरि आदर र सम्मान छ
यो जुनीलाई यस्तै भयो अजम्बरी रहनेछ

मायाको भाव झल्किन्छ तिमीलाई देख्दा
माया सधिं ताजा हुन्छ तिमीलाई भेट्दा

कस्तो कस्तो हुँदो रहेछ मन भरिएर आयो
धेरै पछि देखे उनलाई आँखा भरिएर आयो

दोउदिएर उनलाई हेरी एक टक
असेट मेटिने गरि अंगालोमा भरिन मन लाग्यो
किन होला भाग्य पनि यति निष्ठुरी भाको
छुटाएर जिस्काएर के परिक्षा ले को

Bichara haru life ko kati event haru holan. Kati yakantama royo holan xutepaxi. Xutnu aa aafnai problems thiyo hola. Ani xutyo. Yasari sangi dekhda khusi लाग्य

तिमि संग छुटे पछि आँखा हरु रसाई रहे
बिचरा भन्ने हरु भित्र भित्र हांसी रहे
कति मिठा  कति तिता पलहरु सबै आए
तिनै पलहरु सबै सुन्दर लाग्दै गए
एकान्तमा एक्लै बसी कति झरे आँसुहरु
जिन्दगीको भोगाईलाई लेखुं कसो गरि !!!

जिन्दगीको बिभिद परिबेशमा भेटेका छुतिन्छन, छुटेका भेटिन्छन, नभेटेका भेटिन्छन तर आंफ्नो खै के हुने हो थाहा छैन !!!

संगीत यस्तो चिज हो हरेक परिस्थिवश छुटेकाहरु भेटिन्छन र टुटेकाहरु जोडिन्छन

 संगीत से रुलाभी दे ता है संगीत से मिलन और नई उचाइको और भी उचाययोमे पूहुचने की तागत और हौसला दे ता है ।।


मान्छेले गाई जस्तो चर्ने होईन

बिहान  बिहान  उठ्यो  खुखुराको  बासले
कस्तो  रमाईलो  भो  यौ  मन

Day39- AWS - Storage Gateway, FSx, Luster

 Class Notes

Session 39-AWS-Storage Gateway/FSx/Spot instances


On prem
- Our database server is running on our datacenter.
-
mySQL
- /var/lib/mysql

scale ---->

IOPS
- Throughput
- Parallel processing


WebServer
- /var/www/html


Storage
- object
- file
- block


# systemctl start httpd
# systemctl status httpd

# curl 127.0.0.1

s3 -> scale to PT or more ... almost unlimited ..

we will take a space from s3 to local system
- but we mount in such a way that data stored on local system, will be stored in s3.

filesystem   
blockstorage

- We will use nfs share.

bucket storage converted into filestorage.

connect cloud storage to local storage (gateway)


On prem server
- Mount a directory (From aws)


NAS (nfs)/SMB
SAN (iSCSI)


s3
- bucket

local server <---  NAS/SAN <--------S3

 

 
Storage appliance
- Hardware
- virtual



1. Login to aws console
- search for storage gateway ( hybird storage)

2. - start
3. - select gateway type
a. file  --> select
b. volume
c. tape

On our on prem,
we want to get NAS(nfs) storage. -> storage gateway (file.nas)

gateway is a virtual system which has os, cpu/mem

4. Seect host platform (5 options)
- vmware esxi
- microsfot hper-V
- Linux KVM
- Amazon EC2  --> Select this option
- Hardware appliance
--------------------------------------------------
We are going to use AWS EC2 instance to install Gateway.
- Since we don't have any other env, we will be using ec2.
Think EC2 instance as an on prem environment.

click on amazon ec2

review the instruction for ec2.

instance type -> m4.xlarge (select it) Note: its not free.

select instance type m4

next -> you need public IP.
next -> storage -> this instance works as a storage gateway.

add harddisk : 200GB -> type general purpose ssd(gp2)
never delete

next -> tag
name: SG_Virt_ec2_NAS
next -> allow all traffic (Security)

port 2049 allow /add
next next finish

our storage appliance is installed, now we have to configure it.
-----------------------------------------------
Get the IP of the ec2 instance.

Now, go back to Gateway page where you had select host platform

click next


service end point
- end point:
select public
next - connect to gateway
specify IP address ->
connect tto gateway


next specify gateway name from the drop down
select time, and other stuffs
and click on activate



Gateway health log group -> disable

save and continue

now, you have your own storage gateway on the cloud. Think this is running on prem.

this host provides storage gateway features..

as of now, we don't have any block or san storage to share

Go to your storage gateway
we create this gateway as file type.
it can only supports NAS.

click on create volume - it will fail. you have to create volume type
for that click on create gateway


now, open a new aws console and create a new bucket
bucket name: lwgatewaybucket

disable public access
and create bucket.

copy the bucket name and go back to gateway

click on file share
configure file share setting

fileshare name: lwgatewayshare
select gateway name
next

storage class :s3 standard
next -> summary - review

click create

now to back tp gateway -> you will see 1 file share

this storage gateway is configure as gateway server.

now, we will mount the s3 on local system.

click on file share -> status : available

go all the down on the button, you will see the command to mount. coy the command

now, you can go back to your local system abd login

run df -h and review the ouput
now, run the command you copied earlier.

# mount -t nfs -o nolck,hard 172.31.42.94:/lwstgaewayb1 /var/www/html

you have to change private ip to public
go to the gateway, copy the ip
# mount -t nfs -o nolck,hard 15.34.54.23:/lwstgaewayb1 /var/www/html

# df -hT
review the output
size you see is 8.oE -> 8 exxabyte size.

cat > /var/www/html/index.html
hello, welcome to clob

data is created and everything is sync to s3

for SAN
go to storage gateway
-> create a new gateway and select volume gateway
 two options
*cached volume
*stored volumes

select cached volume (low latency) [use two disk on efor buffer and anoher for data] for speed and performance
next
select the host platform
-> use ec2 instance type

again new instance
M4 -x.large
next next -> harddisk ->

select two disk size 150GB

review the docs.

Tag: name: SG-Virt_SAN

next-> finish

we created an EBS disk. system creates snapshot of ebs disk which is stored on s3.


----------------------------------------
Review
lets say we have two systems and one system is providing storage to the client.

A ----iSCSI ------->    B
server        client

server a provides storage to B.
server is target and client is initiator

SG will be target and it is getting storage from s3.
client is our local system.

-----------------------------------------------------

now launch an EC2 instance -> default everything
tag: san: initiator

allow everything -> next -> next and finish



configure gateway

after selcecting host platform -> select end poing
degind gateway
specify the gateway name
sg_SAN

activate
wait for a while while activiting
click configure loggin

(you see note: you need 150GB for buffer)

no loging - >

-> finish
now, click on create cvolume on gateway page
gateway: sg_SAN
capacity: 10
volume content
iscisc target name: lwsttarget
click on create volume

chap authintication: skip

now, we created one harddisk

this harddisk, connect to client

go to client system (local vm, on prem)
here you will attach harddisk

# fdisk -l
list you all disk on the system.

how to use iscsi
1. discoverry name
2. login


click on the iscsi and you will see a instruction at the bottom of the page

just install the softare
iscsi initiator
/service iscsid status
/etc/init/d/
after software install, discover

iscsiadm ... you need IP as well

from client, we are discoverying ..

iscsiadm --mode discovery --type sendtargets --portal 172.31.38.164:3260


....
follow the instruction

# cd /dev/disk/by-path

you will see the disk here
# ls -l


to delete: go to action and delete

Practice this lab twice, imp

store data on the cloud
- encription/scalibility is managed by aws


------------xxxxxxxxxxxxxxxxxxx---------------------

New topic

FXs
- idea here is windows server, desktop, they have different services. dns, active directory, web server, distributed services

Win ->DFS(like NFS)
- optimized for bigger data

search for 'windows server dfs'

how does microsfot dfs works?

windows complete filesystem is managed by aws called FSX

HPC - high performance computing

search for linux hpc server

FSx also available on linux
linux cluster
- luster for linux system

go to aws console and search fsx
read doc
select FSx fow indows
- click next
specify name: myfsx1
ssd
specify storage capacity: 100GB
specify the throughput capacity: 2048 M/s

Note: its not free service)
go down to windows authentication
- aws managed microsoft ..

.leave as it is and click on next

for linux you select FSX for luster
choose through put speed ..

specify storage capacity: 2.4 tb

rest of the options are same.