Day5 - Docker, swarm, k8s intro

 2/02/2022 - class notes

docker, service, network - overlay, 
Recap from yesterday
If you running muclitple containers, how do you connect them?
https://github.com/fgitsolutions/docker
$ cat docker-compose.yml
version: '3'
services:
  web:
    build: .
    ports:
     - "5000:5000"
  redis:
    image: "redis:alpine"

- Discussed about Dockerfile
- Docker-compose
  - multiple containers
- Single host/standalone host
- If you run multiple containers, you will be out of memory
- Distribute the containers among nodes
We need custer managerd by
 - manager
 - Worker nodes
Who is going to provide this kind of facilities
 - Docker SWARM
 - K8s
 - Mesos
Swarm architecture
docker swarm init
docker swarm join
google and see the swarm internal atchitecture.
- internally swarm manager has multiple components.
 - API -> accepts commands and creates service object
 - Orchestrator
 - Allocater
 - Dispatcher
 - Schedular
Worker Node
 - worker 
 - executor
LAB
Login to your VM
# ssh swarm-manager
$ docker nodes ls
$ docker swarm init  # initialize
 initialized current node is now manager.
you are manager.
@worker node
$ ssh swarm-worker
$ sudo -i
$ docker swarm join <key> ip:2377
Go back to master and run
$ docker node ls
manager is a leader and other is a worker node.
You can execute your program
Docker Service => docker run
Docker stack -> docker-compose
you can execute as a job or a service.
# docker service
read the output
$ docker service create --help
# docker service create --name=n1 -p 80:80 nginx
name -> n1
-p -> port
image-> nginx
List it now,
# docker service ls
# docker service ps n1
scale
# docker service scale n1=3
it created 3 replicas. 
# docker service ps n1
Your service will load balance.
# docker inspect <ID>
it is going to use overlay network
# docker network ls
see the ID and the driver type.
It will creates tunnel between all the nodes
VPC or VNET pairing?
- able to communicate between one network to another. Establish the communication between private network.
# docker inspect ingress
look at under peers, you will see two IPs.
peer mean they both can communicate. its like creating tunnel between these two node. 
manager/node can transmit the communication.
google "vpc peering"
google for "overlay network"

# docker swarm --helo
# docker swarm join-token
# docker swarm join-token manager
# docker swarm join-token worker
# docker service ls
# docker serviec ps n1
how to run muitple cotainer 
# cat docker/docker-compose.yml # refer the example from yesterday


$ cat swarm-compose.yml
version: '3'
services:
  web:
    build: .
    ports:
     - "5000:5000"
  redis:
    image: "redis:alpine"

# docker git pull
# docker service 
review help
# docker service rm n1
# docker stack services mystack
# docker stackk
# docker stack rm mystack
if you want to deploy
our instance is t2.large
# cat ci.yaml
it will create multiple containers

# docker stack deploy -c stack-ci.yml costack
stack is multiple 
List all services from stack-ci
# docker stack services cistack
# docker stack services cistack
if you keep creating, we may go out of resources.
# docker logs <id>
# docker stack service cistack
# docker ps -a

# docker stack deploy -c stack-ci.yaml cistack
Remove
# docker stack rm cistack
# docker stack services cistack
everything is gone.
# docker stack service cistack
# docker logs <docker_Id>
getting error for dogs
# docker stack rm cistack
# docker stack deploy -c stack-dc.yml mystack
# docker stack services or ls
# docker stack services mystack


version: '3'
services:
  web:
    image: 'devopsjuly22017/web:latest
..

# vi stack-dc.yaml
# docker stack deply -c stack-dc.yml mystack
# docker stack service mystack
docker is not production ready by itself. because of the nature. 
docker swarm does not have features that k8s offers.
k8s is production ready. Swarm can be used in production but it does not have alots of features that k8s offers.

Read about "ECS cluster in AWS" - aws implemented docker cotainer (microservice service) implmentation.
go to aws
search for ecs
-> get started

k8s ->
google and read about k8s features.
will discuss 'architecture and internal components;
google k8s architecture
----------------------------------------------------
user (cli) -> api -> k8s master -> Node1|node2|node3|..|Noden  --> Image Registry

you are going to use,
k8s-client
k8s-master
k8s-nodes
k8s features
- open source systems for automatiing deploying ....
- Automated rollouts and rollbacks - you created 3 replicas, version is 1.1 and need to update 1.2. How do  you do it? but k8s will do roll back/rollup. one by one update/one by one update. high availibity. if something goes bad, if will roll back. 
- storage orchestration - automatically mounts storage system of your choice such as aws, gcp or nfs, iscsi, gluster, ceph, cinder or flocker
- automatic bin packing
- service discovery and load balancing
- secret and configuration management.
- horizontal scaling [ up and down your app]
- Batch execution - run jobs
- IPv4/IPv6 dual-stack
- Self healing - restart if fails.
Node level and container level.
Deployment types
- blue green 
- 
- 

Kubernets components
--------------------
Control plane
kube-apiserver
- api server the component of the k8s control plac=e that executes the k8s API. The API server is the front end for the k8s control plane. its like a front desk
- ETCD - database storage (NoSQL database). key value backing store for all cluster data. maintains the state of your cluster. what is connected, 
- kube-Scheduler -> when request comes to say create container, checks which node is free, which node is capable to create the conteiner.  load balances. 
kube-controller-manger (internal component)
  - control takes a decision, logically each controller is a separate process 
some types of controllers
- node conteoller - checks to see if nodes go down. checks health of nodes.
- job controller - watches for job objects. self healing, creates pods to run jobs.
- end poing controller
- service account and token controllers

@aws, create auto scaling group

cloud controller-manager
- this can use use with cloud service providers.
- cloud specific controller logic.
node controller
route controller
- service controller
5 components
- API server
- 
Node components
on node side,
1. kubelet
- an agent runs on each node in the cluster. makes sure that containers are running in a pod.
kubelet does not manage containers which were not created by kubernetes.
2. kube-proxy
- kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the k8s server concept.
- it maintains network rules on nodes. These network rules allow network counication to your pods from network sessions inside or outside of your cluster.
- it uses the operating system packet filtering layer if there is one and its available, otherwise kube proxy forwards the traffic itself.
3. Container runtime
- container runtime is the software that is responsible for running containers.
- k8s supports containerd, cri-o, docker, rkt and other implementations. 

apart from these components, we have addons
- DNS
- WebUI (dash board)
- Container repsirce monitoring
- Cluster level logging

There are 
- Opensource
- Enterprise :- ~PAAS
k8s setup
google, kubernetes set up

create minimum 2 server
1. install kubeadm, kubelet and kubectl
2. Initialize run commands kueadm init <args> - initialize master and provides the join command.
we are going to use two types of commands
1. client command - kubectl
2. Admin : kubeadm (master)

github.com/qfiitsolutions/k8s
https://github.com/qfitsolutions/k8s
know little about openshift service as well.
https://docs.google.com/document/d/1_ZQ8XN1dfcaXkBJHPMIlZqoW4_BePbrCU_LW3yvDTAs/edit

cluster contains
- master
- nodes
Tomorrow's topic
- set up k8s
- PODS
= Write yml
- Dashboard
- Deployment
- Service

================================

apiVersion: apps/v1

kind: Deployment

metadata:

  name: ais-ifm

  labels:

    app: nice

spec:

  replicas: 1

  selector:

    matchLabels:

      app: ais

      tier: web

  template:

    metadata:

      labels:

        app: ais

        tier: web

    spec:

      containers:

      - name: tomcat

        image: tomcat:latest

        ports:

        - containerPort: 8080

        resources:

          limits:  

            cpu: 1000m

            memory: 600Mi             

          requests: 

            cpu: 500m

            memory: 300Mi

      - name: nginx

        image: nginx

        ports:

        - containerPort: 80

        resources:

          limits:  

            cpu: 400m

            memory: 200Mi             

          requests: 

            cpu: 200m

            memory: 100Mi

Day4 - Docker , docker swamp, docker compose

Day4 - Docker  - class notes
Docker Network
  - Bridge
  - Host
  - None ->
Volumes
  - Mountpoints
  - Persistent data
How to run docker
  - as a job
  - As a service
Ports
  -p
  -P
Image -> From running containers
Push -> Registry
Docker Hun
ECR
Quay
Jfrog artifactory
Nexus repo
Image creation
Image: docker commit containerID
Developer code/release/microservices
  - Avoid manual process
SHould be able to perform
  - pull
  - Run
  - copy
  - configure
  - save (commit)
We want to avoid these above step perform not manually but automatically. How can we do it?
- By using image build as a code concept. 
We use Dockerfile and specify all the stuffs there.
Dockerfile is going to help us to build the image. We will specify the base image, specify the packages needed. what files you need to cpoy, what ports, what env, what packages you want to download, what script do you want to execute.
Docker file contents,
  - Image
  - packages
  - copy, download, copy files, certificates
  - expose (ports such as 8080, 5000)
  - Env
  - initscript
Dockerfile
FROM ubuntu # what image do you want to specify, here we specify ubuntu, so it will download ubuntu image
RUN apt install default-jdk* -y # install jdk packages
ADD . /app   # copy everything from current directory to /app on the container
WORKDIR /app  # /app is going to be home directory for the user login.
EXPOSE 8080  # exposting the port 8080
CMD [java -jar abc.jar]  # start a service, when system boots up, service starts automatically. Run initial command. (You can call it as a boot strap script)
so the dockerfile content is as floows
# cat Dockerfile
FROM ubuntu
RUN apt install docker-jdk* -y
ADD . /app
WORKDIR /app
EXPOSE 8080
CMD [java -jar abc.jar]
$ docker build -t web . # [ docker build -t (tag)
when building the image, it will install ubuntu and at the build time, it will install something (jdk in our case) at the time of build. Run command helps
cmd command runs the java -jar abc.jar.
-----------------------
LAB
# start your VM, and login to the VM.
github.com/qfitsolutions/docker/

FROM python:3.4-alpine
ADD . /code
WORKDIR /code
RUN pip install -r requirements.txt
CMD ["python", "app.py]


git clone github.com/qfitsolutions/docker.git
FROM python:3.4-alpine
ADD . /code
WORKDIR /code
RUN pip install -r requirements.txt
CMD ["python", "app.py"]
========================
open new session and monitor
# docker ps -a  # watch, 
===========================
# docker --help
# docker build --help
-t - tag
-f - name of the file path
# docker build -t web:latest .
. means it will refer the dockerfile from the current directory.
observe the output. 
dockerfile is analyzed and proceed...
it performs 5 steps,
- it is pulling the image
- added the content. /code
- running in
- removing intermediate container (docker build command when downloaded the image, starts a container, add content, and it removes it self,
it starts a new instance and starts next step -> app.py
installing packages, removing .. just look at the output. 
# docker ps -a # command shows the imtemediate container. keep running
Run the commands inside requirements.txt 
# cat requirements.txt
flask
redis

Review this file,
https://spring.io/guides/gs/spring-boot-docker/
# cat Dockerfile
FROM openjdk:8-jdk-alpine
RUN addgroup -S spring && adduser -S spring -G spring
USER spring:spring
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

Google,
Entry-point docker example
# cat Dockerfile
FROM ubuntu-trusty
CMD ping localhost
if you build this image, it will give you ping output
# docker run -t demo 
# docker run demo hostname
# docker run --entrypoint hostname demo
here hostname was run in place of ping command. It is not executing ping command but hostname is overwriting the default command.
so default entry poing is replaced.
$ docker run --entrypoing hostname demo
overwride the default cmd command.

# docker run -d demo
# docker ps -l
# docker exec 

FROM ubuntu:trusty
cmd can overwide default behaviour.

CMD you can override, on ENTRYPOINT, you can't overwride.
Google for
Dockerfile example
https://docs.docker.com/get-started/02_our_app/
https://docs.docker.com/samples/dotnetcore/
There are multiple repositories, modules available.
When you have multiple micro services, you have single application, you have multiple microservices, how do you start whole environment. 
Even, on dev, Test,Prod env?
Microservices
- Build all images


Google microservice architecture
https://microservices.io/
https://microservices.io/i/Microservice_Architecture.png
on the image above, there are 7 microservies.
4 micro-services have 3 dbs, different images.
Here, we are going to run Docker run command.
we have to run 'Docker run' command multiple time.
we will define image, run docker run multiple time and variable img
such as 
Array:[img1 img2 img3]
For loop:
  Docker run $img
but the problem is say we will us port, network, volume, name or more.
Some case some option may needed and some may not needed.
it becomes complecate.
rather than that, we use yaml code.
we will specify dynamic input
for eg,
nexus
 img
 port
Jenkins:
 img
 port
 Volume
 network
Maven:
 img

So, what is yaml file?
its like an xml file. 
xml, json, yml -> these all keep data. carry data.
we have Map and list
in yaml, we us map and list
kep pair
Map:
  Key: value
  course: devops
  City: DC
List # multiple values, not only key pair, muitiple values
  Cities:
   - DC
   - NY
   - LA
Fruits:
  Banana
  Apple
  Orange

List, specify multiple values...
services:
  jenkins
  nexus
  maven
  spring

services:
  jenkins:
    img: jenkins
    port: 8080
    network: mynw
  nexus:
    img: nexus
    Port: 8081
  maven:
    img: maven
  spring:
    img: mysprint
    Port: 8443
Docker will read it.
We will use docker-compose can read this yml file. 
docker-compose -f abc.yml/docker-compose.yml up
 
to start multiple containers
$ docker-compose -f abc.yml/docker-compose.yml up
 
How can we execute?
login to your system 
# cat docker-compose.yml
version: '3'
services:
  web: # web application, which we name as 'web'
    build: . # build a new image, 
    ports:
     - "5000:5000"
  redis: # backend application, pulls the image and builds
    image: "redis.alpine"
# docker-compose
no command found
apt install docker-compose
we need to install it. 
# apt install docker-compose -y
# docker-compose
you will see options
you see, start/stop, rm, run, build and much more options
# docker compose -f docker-compose.yaml up -d
up is sub command
-d -> run in background, deatach mode.
it will successfully builds first and after 2nd service.
two containers at a time.
# docker-compose ps
rather than docker run, use docker-compose command.

ip:port
if page is not displayed, 
go to instance, security group and add 5000 port - anywhere.
and refresh the page, you should be able to see the page.
======================================
try this one too and see what happens.
# docker run -itd --name web --image redis:alpine -p 5000:5000
docker is used to run single container at a time where as docker-conpose is sued to run multiple containers at a time.

- docker swarm
- Kubernetes
- Mesos
- tanzu
docker swarm is implemented by docker while k8s is by kubernetes by google.
google docker swarm atchitecture.
- internal distributed state sotres
- Manager manager manager
worker worker worker worker worker
its very simple to bring it into 
there are few commands
# docker
look at under management commands.
# lets make current machine as a maanger
# docker swarm
read the output help content.
# docker swarm COMMAND --help
# docker swarm init
it initilizes.
current node is a manager.
to join other nodes.
$docker swarn join --token <token id> 192.168.10.20:2377
Create a new aws instance as swarm-worker node
use t2 large to create new instance.
install docker command
$ apt update
$ apt install docker.io
run the join 
$ docker swarm joing --token <tokern no> 192.168.10.20:2377
you may get error, 
add inbound rule and add port all traffic / all traffic
now should be able to work without issue. if you get error, keep adding the error based on the error or take the corrective actions.
# docker swarm leave
# rejoin using the above command - docker swarm join...
# cd docker
# vi docker-compose.yaml
web:
  image: "devopsjuly22017/web:latest"
  ports:
   - "5000-5000"

# docker stack
# docker stack deploy -c docker-compose.yml mystack



# docker node ls
read about k8s at kubernetes.io

service
x

Maven, Jenkins - Class Notes

 Maven - Class Notes
----
apt install tree
Build configuration tool - Maven
Project directour structure
Build process (pom.xml)
Life cycle phases
  - validate
  - compile
  - Test
  - Package
  - Install
  - Deploy
command
$ mvn -f pom.xml clean install/deploy
install vs deploy
nexus/jfrog

draio - mac drawing tool
diagrams.net

maven -> jenkins -> nexus repo
webhook (jenkins)

--------------------
Building lab environment
- login to aws console
hostname: maven-host
- connect 
$ ssh -i "user.pem" ubuntu@ec2-192.168.10.20.us-webt-1.compute.amazonaws.com
security -> 
inbound rule
ssh -> use your ip
ssh anywhere 0.0.0.0
or ssh source anywhere -> specify your ip

$ tree ../core-app
$ mvn clean
going to delete target directory
when you run mvn compile/package
the output, packaged file will be on target directory
$ cd core-app; ls
$ mvn ..
it generates target file 
$ mvn clean install
deletes all old target directory contents and re-creates
it compiles, packages and stores on repo as well


multi module projct
$ cd maven-multimules
$ tree simple-parent
 -src
   - main
   - test
$ git remove -v
$ cat pom.xml
distribution manage task, we are going to manage
push to remove site using vscode
$ git status
$ touch .gitignore
$ vi .gitignore
target/
$ git status
$ rm -rf .gitignore
$ mvn clean
$ git pull
$ mvn clean install
$ git status
Notice: git will ignore the target file.

We have to login to
- ssh: ec2
- git we use, clone/pull operation
- we use Maven
developer modify the file everyday.
so you have to manually upload/download
so, we use process continue integration tool (jenkins) to automate the task.
We are going to install jenkins tool.
continous integration tool
- jenkins
- gitlab runner
- circleci
- azure devops pipeline
- bitbucket pipeline
- gitub actions
- code pipeline
- travis
- teamcity
=========================
4 steps
Step 1. maven commands
Step 2. jenkins
Step 3. 
Step 4. 

You can install maven and jenkins on the same system but its better to use different instance.
1. Launch a new instance - use t2.small for jenkins
2. t2.micro for maven 
login to the hosts:
Install jenkins
use online instruction
set up repo or download the package and install - either way..
$ sudo apt-key add - 
$ wget -q -o - https://pkg .. > /etc/apt/sources.list.d/jenkins.list'

$ apt-update
$ apt install jenkins
$ apt repolist (apt ubunto repolist)
failed to start LSB: start jenkins at boot time..
read the error message..
no jave executable found in current path

Install JAVA
look at the bottom of the page, you will see guide how to install.
$ install openjdk-11-
$ systemctl start jenkins
$ systemctl status jenkins

get jenkins public ip and paste on browser
http://<ip>:8080
you may not get any page, so you have to add 
click on the instance
- security  and add port 8080 anywhere and save the rule
refresh on the browser, you will see it.
follow the instruction to set up user
jenkins home directory
$ cd /var/lib/jenkins
you finally completed jenkins server.

click on the create a job from the middle of the page
item name: test job
select Freestyle projecct -> basic type 
click ok
you are on general
select discard old builds
source code management
git
build
add exxecute shell
echo "Hello world"
click ok
we created a simple job.
click o build now, 
click on build history
- click on console output - you will see the output
- go to command prompt and run $ echo "hello, World" and compare the output.
as of now, we created new item, free stye project. selected couple option with echo command.



$ git remote -v
get the url
now go to source management and paste the git url
brnach /master
save it and build now.
it will perform the job.
job is executed. 


to to home page
manage jenkins
- manage plug-ins

# cd /var/lib/jenkins
# cd workspace/testjob
# ls
src
pom.xml
where do you define the linux credentials?
- we install jenkins on the linux server so it use the system account to launch the job.

click on manage plugins
search "maven
select maven integration
go to home dir
- click on new item
Now, you can see maven project
name: core-app-maven-build
select maven project
clcik ok
copy your repo 
specify git
go down
build option , it automatically understands
Root pom
pom.xml
Under goal and option
clean install
save the job
you save new job
to go home, amd click on manage jenkins
- configure tools, global tools configuration
click on add maven
name: maven-3.8.4   # specify the path
select Install automatically
jenkins automatically downloads maven pacakge and installs it.
you can select the version you want..
once saved, click on build now.
click on the job and console output
where can we put maven server details if we jenkins and maven are on different server..


As of now, we install jenkins and created job. and successfully executed the job
check git link for 1:05:00 - 1:07:50
$ mvn archetype...
$ tree core-app
github.com/devopsgsvc

add multiple people into the team and distribut the job.
that way, you can add number of executors
- click on manage jenkins
- Manage nodes
to add plugins
- add, remove disable enable plugins..
- you can add n numbers of hosts.
- click on new node
or
configure clouds
node name: node01
select permanent agent
description: testing
number of executor: 3
remote root directory: 
paste from the command below
go to command line and create dir
$ mkdir jenkins-agent
$ pwd
copy the path

Labels: maven-label
Usage: use this node as much as possible
launch method: launch agents via ssh
Host: IP address - if cloud, use provate IP
credentials: 

Host key verification strategy: known hosts file verification strategy
to login from node1 to node2, we can use password or key.
Node 1 ---ssh---> Node2
$ ssh node2
you can use id_rsa.pub per user basic or
if you are using cloud, you can use authorized key from user's home directory to .ssh.
This public key can also be used to login to remote machine for passwordless login.
add credential
domain: global credential
kind: ssh usernane with private key
id: ubuntu
private key:
key:
paste the output of your authorized_keys value.

Host key verification strategy:
save
click on node01 and click on log
you will get the status if it is succesfully connected.

why labels
if you want to add multiple node, each node may be doing different tasks, say 2 node maven, 2 docker, or other job, so label will help you to pick up..
jobs -> configuration
restrict where this project can be run 
  label expression: maven-label
specify your label here.
once done, click on build now.
click on job# and click on console output.

click on workspace - delete
or go to command line and browse to the jenkins location.

go to home
- manage jenkins
- configure clouds
error - no cloud implementation for dynamically allocated agents installed .. go to plugin manager...

Do some research on,
how to configure cloud concepts on jenkins... 
configure -> fill required info, 
Will cover this instance tomorrow..
- dynamic node
- Webhook
- Authentication
- Pipeline
- Nexus
- Docker
If you are using cloud service, make sure to stop/delete.

draw.io

Day2 - Jenkins

Jenkins - review - Class Notes

Lab: Create a jenkins server
- Create job
- execute job
- Check log
- Create a node and add to the server
- Webhooks

Install http server java
https://stackoverflow.com/questions/3732109/simple-http-server-in-java-using-only-java-se-api

=======================
Recap
- Maven
- Job
- Jenkins
  - Jekins jobs
    - Config )restrict this project can run
    - URL 
    - Where job is executed
    - Added new agent (node01)
    - Manage jenkins (manage nodes and clouds)
      - nodes - configure - add node and add host with credential (private key)
    - job execute
    - click on job
    - play around

init scripts
--------------------
item name; test pipeline
Strategy: log rotation
15 days
pipeline
pipeline script
hello world
script:
pipeline {
  agent ..
}

copy this code and go to vscode
and search jenkins and paste the code.
if you want to have miltiple stages, you can paste for n number of time and change stage hello1, hello2 and so on.
You can copy this code to jenkins and save.
Click on build now to execute the code.
You can also select jenkins + Maven.

jobname/pipeline syntax-> 
How to trigger jenkins job automatically.
How to access nexus repo?

click on job
- configuration
- build trigger
review the options
build periodically - its like cron job. batch type job. specify the schedule..
- trigger builds remotely (eg, from scripts)
  end point URL
jenkins_url/job/core-app-pipeline/build?token=sdfsdafsdfasf
http://ip:8080/job/core-app-pipeline/build?token=sdfsdafsdfasf
note: only login user can perform the build job
so you have to use the credentials.
- you can't use password but can use token.
click on API TOken
add new token / generate
http://user:token@job/core-app-pipeline/build?token=sdfsdafsdfasf
go to jenkins
- settings
web hooks
 add
payload URL
what time you want to trigger, review the options 
click on add webhook
job is automatically trigger.
=========================
launch new instance
- free tier
- t2.medium
- configure - add storage , security group port 8080, 80 ..
name this instance as nexus
go to security group and add a port
cutom tcp 8081 anywhere, 0.0.0.0/0

go to instnaces and click on nexus. verify its on running state. 
- connect - copy key 
how to create nexus repository
------------------------------
jar file generated will be uploaded to repo.
two approach
- normal approach
- docker hub
install docker package
$ apt install docker
$ apt update
$ docker

search nexus
sonatype nexus docker ..
$ docker run -d -p 8081:8081 --name nexus sonatype/nexus3
you can also use jenkins.
$ docker ps
$ copy nexus public ip address:8081
you got nexus repo
login: admin
pw: get from below command output. 
$ docker exec -it nexus cat /nexus-data/admin.password

it will promt you for new pw, 
- click next - finish

now, we have to upload our artifacts. 
- click on repository
- you can create or use existing repo
- click on repository
you have lots of options available. select based on your request.
maven has 3 type.
proxy - proxy to your repo. 
maven2 type
name: core-snapshots # backup option
layout policy : strict
blobstore(store): 
hosted: deployment ..
..
create repo

maven2
name: core-releases
other option not allowed..
disable redeploy
click on repo and you can copy.
Now, you can go back to your git and change the corporate repository.
<id> maven </id>

you can also take a snapshot version


=================== 
on nexus
Security
- proviledge
- roles
- users
- annonymous accesss
- LDAP
- Realms
- SSL certs

Roles -: role id: core-deployment-role
name: core-deployent-role
role description: core deployment
create role
and create users - create local user
- core user
email: test@best.com
passwod: status: active
done..
to upload aotumatically, use anonymous access...
setting.xml
maven 
github.com/devopsgsc/core-app/tree/dev
today
dynamic agent
webhook
nexus
- setup -> docker
- repo
- perissions
- settings.xml


54.177.243.80:8081
today

Day 2 - Jenkins - Class notes

1/13/2022
condemned

https:///github/com/hshar/website.git

Jenkins day 2
-----------------
CI/CD pipeline
1. Commit change
2. Trigger build
3. build 
4. Notify of build outcome
5. Run tests
6. Notify of test outcome
7. Delivery build to environment
8. Deploy where necessary
----------------------------
1. Login to your jenkins
http://192.168.10.20:8080/

Get the passowrd
# cat /usr/lib/jenkins/secrets/jkAdminPassword
Use this password to login
Customize jenkins
- suggested plugins
  These plugins will help you to work on it later. 
Getting started
Create first Admin user
Note: Make sure to add 8080 port on your inbound rules.
click on start using jenkins

===================================
Intregation with jenkins
http://github.com/anujdevopslearn

go to repo and click on fork button
you can get source into your system
open java based source code
search: maven
MavenBuild

cLICK on new item -> : 
Freestyle project/Multi-configuration project are lagecy one..
after 2.0 launch - folks stop using these. thats why these are not recommended.
Thats why jenkins came with idea of pipeline. Its a feature that developers are using it which can be use for pipeline. Even if you loose the jenkins, no backup or nothing available, you don't have to worry about the config lost. 
You can recover your jenkins job.
sample groovy code
glst.github.com/saltnlinght5/3756240

basic scalaton of jenkins code
node{
  stage('Checkout Code') {
  }
  Stage('Build Process') {
 
  }
  stage('Development'){
  }
}
Put this code into jenkins job
Enter an item name: BuildAutomation
select the pipeline

node(slave){
  stage('Checkout Code') {
  }
  Stage('Build Process') {
 
  }
  stage('Development'){
  }
}


Click on pipeline syntax
- automate job generator
you can select the options and generate code
- select  



========================
NuGet.org/packages
you can search for different packages.
nuget package config
for dot net technology

Day1 - docker intro, Jenkins

Day1 - docker intro, Jenkins - Class Notes
1. Login to aws comsole
- select your region
- 
Recap 
- understanding mavin
- Jenkins
- Jenkins agents
- Pipelines
- nexus repository (use to store artifacts)
 jar
war
zip
jekins
nexus
front end app
database
IIS
tomcat
nginx
apache
jetty
how you are going to launch?
- first you are going to create VMs(compute engine)
You are going to 
- install packages
- configure files
- start services
not only this, you are also going to much more such as,
- User/group cretion
- certificate and much more 
----------------------------
load balancer
 - Node Pools
Launch configuration (init scripts) -> Custom AMI(comes with all required packages)
- cloud watch (notification)
- scale in/ scale out

Install one VM 
- install all required software
- You can use podman to create image or use other tools.
=================
VM - using high memory/cpu
WHat is the cause?
You look for different aspect such as 
- AMI - Virtual machine architecture
Root cause
PM vs VM
physical vx virtual machiens
- Multiple OS
- Use only one - one at a time
- partition CPU/Memory
We want to use multiple machines simultaneously. how?
- Front end as well as backends.
- We want to partition same machine half and half for backend and front end.
Instead of physically divinding memory, we want to divide virtually. 
This concept is called virtualization.
Virtualization is the process of running multiple virtual systems or resources on top of a sysge physical machine. These resources could be a storage device, network or even an operating system.
-> app
-> Virtual Machine [ VMs (Guest OS)]
-> Hypervisor
-> physical
Problem with VM?
- Platform specific (windows or linux)
- resources utilization if not good
- In a system physical system, you have systema and systemb and you have two application running but cpu/mem used is only 20%.

There are three components.
google : vm architecture
Host OS
Hypervisor
Guest OS
Bin/Libs
Application

There is a concept of havy weight to light weight. What is taking heavy weight out of these component.
Here, guest OS is taking tool..
physical memory is a problem, 



what is the solution?
virtualization on OS..
- it will become light weight
it will only contains
- lib/lib
- application
container technology/containerization
OS level virtualization
- light weight
- Dynamic memory allocation
app1   | app2     | app3 ..
Bin/libx  | Bin/libs | Bin/Libs
Container engine
Operating system
Hardware

now, it became light weight. we can create image of the instance.
This image is going to be
- Dynamic memory
- Faster bootup = with in sec
- Simple network to communication
in sinlr host, you can create multiple conatiner.
using these options, you can isolate software isolation.
You have Dev/Test/PreProd/Prod env as well.
Environmental differences
- Traditionally, we may have different image but now, you will use same image that is created using the instance in.

Note: Once you create image, you can not change the software.
- Isolation 
- One image for all env
Container engine
- Docker engine
- CRI-O
- RKT
- Containerd
- PODman (utility)

google docker architecture 
docker.com/get-started ..
Three components
- Docker clients (utility command) - provide instruction - build image, install
- Sends instruction to engine
- Docker engine
- Docker Registry (Storage for image)
Docker runs on client / server architecture.
Docker client/server communicate using rest aAPI

VM
 - Docker package (client+engine)
Rest api (its like http server.)
- when you run make change on code at github, jenkins running the job. 
client connects to rest api (docker daemon) for connunication or to send instruction.

=======================
ssh <instanceIP>
# docker
no command found
# systemct status docker
# apt install docker.io
This package is not available on index list
# apt update
# apt install docker.io
# docker
Repo list location
# /etc/apt/sources.list.d; ls
---------------------
apt-get update
apt-get install jenkins
---------------------
# systemctl status docker
active/running
docker application container engine

# docker
show you help
go under commands
start/stop/pull/push/kill/inspect ..

How to use container? 
There are two ways, you can use
1. As a job - just run one time (execute a single script, see result and exit)
2. As a service - Running 24/7, all the time. 

in a linux point of view,
VM
 - hello.sh
$ sh hello.sh
$ sh hello.py
$ java welcome
you will get output. but you need depeldencies.
Lets see how to use docker as a "job" point of view.
you don't need dependencies since container contains all required packages (but you have to have it right? - lol)

Registry
- docker HUB
hub.docker.com (like git hub for images)
search windows, centos ...
search for ubuntu and see how to install
# docker 
images, ps, pull, run
# docker pull ubuntu
# docker pull ceontos
$ docker images
# docker run --help
see the usage : docker run [options] IMAGE [COMMAND] [ARG...]
-d , -t, -u, 
# docker run -i -t ubuntu pwd 
-i -> interactive
-t -> image
ubuntu -> image name
pwd -> command
# docker ps # no output
# docker ps -a # show all container
it takes couple of seconds to start container. 
# docker run -it ubuntu ls -ltr
you got the output at a single time.
How to run multiple commands
# docker run -it ubuntu bash
# you are connected to container
# ls -l; pwd
# docker 
no command found becuase you are inside container and docker is not installed.
We are on foreground mode and running command inside the container
# exit
# docker ps
# docker ps -a 
you will see the output now.
The container is executed once the job is completed.
but we can run docker in deatch mode.
# docker run -it -d ubuntu
# docker ps
# docker run
# docker --help
review the -d help option
3 ways to run command on docker
-------------------------------
1. run a command 
# docker run -it ubuntu pwd
2. Run multiple commands
# docker run -it ubuntu bash
3. Run in detach mode
# dicker run -itd ubuntu

see the option exec - run a command in a running container
you can execute on ecommand
# docker ps
# docker exec -it <container ID> pwd
create a direcoty on container, and lets install software
# mkdir app; cd app
# touch welcome.sh
# apt install vi -y
# apt update
# vi welcome.sh
echo "Hello world"
Note: Linux container can run on linux and windows but windows container only runs on windows (I am  not sure about it)
# chmod 755 welcome.sh
# ./welcome.sh
Hello world 
# exit
Run the command on docker from host
# docker exec -it <container_ID> sh /app/welcome.sh
# docker exec -it c442bhc54eee sh /app/welcome.sh

# docker ps
review commit option
# docker commit --help
docker commit [OPTIONS] CONTAINER {REPO[:TAG]]
# docker commit <container ID> welcome:latest # we are creating image name: welcome
latest is the version or you can state v01
# docker images
It created your own image..
see the size..
compare the original size and the one you created, you will see different size because its your custom image.
# docker images
# docker ps
# docket stop <container id>
# docker rm <container id>
# docker ps -a
now, its gone..
Now, try it,
# docker run --name=eg1 welcome sh /opt/welcpme.sh
eg1 is the name of your container.
you will see the output
- you can customize it and use as an image.
- whatever you made changes, you can create image and save for later use. You can keep this image wherever you want.

docker hub allow you to store the image. You can sign up for free.
once login, click on repository
create repolitory
name: welcomeimage
private/public and click on create
you just created a repo on docker hub.

Now, look at the option tag
create a tag TARGET_IMAGE that referes to SOURCE_IMAGE
how to rename?
there is certain format image need to be on
such as user/image_name
# docker tag welcome devopsjuly22017/welcomimg:latest
# docker images
How t push it..
# docker push devopsjuly22017/welcomeimg:latest
denied, access denied
you have to login
# docker login
login with docker id
username: devopsjuly22017
passwd: ******
Login succeeded
# docker push devops/welcomeimg:latest
it is pushing
go to hub.docker.com and verify the image is uploaded.

# docker ps
# docker images
# docker ps -a
# docker rm ex1
# docker rmi welcome # use rmi to remove image
# docker rim <image_ID>
# docker images
Now, image is deleted.
Now, lets go ahead and pull it.
get the link from hub.docker.com
# docker run devopsjuly/welcomeimg sh /api/welcome
your image is downloaded as well you get the output.
Next session:
as of now, we executed some commands, next we will bind with port number.
we will discuss about
- service
- network
- volume
- dockerfile
- docker-compise
Assignment:
Create an image from running container, push the image to aws ecr repo.
Hint: aws console - search ecr -> elastic container registry
- create a repository
- get started
and follow the steps

Day3 - Docker training notes, persistence volume (PV)

Day3 - docker info

Containerization
Physical machine vs Virtual Machine vs Container
PM vs PM
PM - Virtualization is enabled on memory lavel.
VM - Virtualization is enabled on OS level
Container
     - Light weight image
     - Isolate the software configuration
     - Easy to ship
     - Dynamic memory
     - faster boot up
     - Simple network
Docker
- Client
- Engine
- Registry

-----------------------
apt install aws-cli
# whic aws
# aws configure
access key:
user-> security credential -> 
get the security key
-----------------------
We created VM and install docker software
- 

Use public registry - docker Hub which you can subscribe for free..
on aws, you can configure using
- go to ecr -> getting started .. next next 

Docker tag - tag the image (version) and push it to registry


docker tag ...

contiiner registry ..
a

aws configure
aws ecr get-login-passwd --region us-west-1 | docker login --username AWs --password-stdin 3763627288.dkr.ecr.us-west1.amazonaws.com
# docker tag d13.... 3626464.dc.de.....
# docker push 98283.dkr.ecr.us-west-1.awsaws.com/avcv:latest

go to ecr -> repository -> avcv 
apt install awscli -y
---------------------
run docker as jobs
$ docker run <imageid> command
job can run at particular time.
Service keep on running
$ docker run image-id
@azure
-> all services 
devops.mar1.2019@gmail.com
container registry -> create container registry

-> create a virtual machine
- create -> virtual machine
- subscription
- resource group
virtual machine name
security: std
image:
size:
adn=ministrator account:
authentication type:
username: docker
pw: ***
in bound port rule:
allow seclected port : 22
leave default
monitoring disable
get the ip
$ ssh docker@IP 
$ apt update
$ systemctl status docker
# docker run -itd --name=n1 nginx:latest
pulls the image and installs after download.
# docker ps
service is running and port is listening.
# docker exwc -it n1 bash
you are on the container with bash shell
# curl -i https://localhost:80
Here, you started a container n1, your service is started and port is listening at 80.
Now, we want to access at the docker level. How do you do that?
by default, docker is going to provide one IP address to this container. a bridge network. 
# exit
# docker inspect n1
inspect meta data of the container. Everything display is about n1.
check under network, you will se ebridge and review the IPAddress
# docker network ls
# docker network inspect bridge
# docker stop n1
# docker run -itd --name=dummy alpine
# docker start n1
# curl -i http://172.17.0.2:80
# it does not find because IP got changed.
# docker network inspect bridge
IP got changed. n1 will have different ip since n1 was stopped.
we can not relie on this ip address. customer may not be able to connect since its changing. 
how do you resolve (over come) this kind of situation?
- you can't relie on this kind of address.
http://172.17.0.2:80
http://localhost:80
expose the port outpide.
instead of ealie on port, we will relie on port. port mapping - port forwarding...
how it works?
inside pod, it will have port and outside also have port number.
such as 80, 8080
like this you can create different container.
this way, you can create n number of container.
let do the majoc
# docker stop n1
# docker rm n1
# docker run -itd -p 80:80 --name=n1 nginx:latest
# curl http://localhost:81
you will get output.
if you try
# docker run -itd po 80:80 --name=n2 nginx:latest
error: port is already on use.
# docker rm n2
# docker run -itd -p 82:80 --name=n2 nginx:latest
# curl lpcalhost:82
# dcoker run -itd -p --name=p3 nginx:latest

# docker ps
# curl -i http://localhost:43567

how do i connect from my pc to the container.
- instead of container ip, we will use public IP address of host machine and the port 
13.145.123.55:81
nothing display? why?
you have to enable the port number.
go to aws -> security group
or 
@azure,
Vm -> networking -> network security group - add in bound rule. 
destination port range *

now go back to the browser and paste the ip address.
you should see the output on the browser.

============
There is another type of network - host
perhost, you want to create nginx container, in this scenario, you can create 'host' type of network.
# docker run -itd --network=host --name=n1 nginx
what it 
# docker ps
# curl http://localhost:80
At the browser
http:IP:80
#
docker run -itd --network=host --name=n1 nginx
# docker ps / -a
# docker logs n1
you can't create host network 
but can create bridge network
# docker network
# docker network create mynw
# docker ps
# docker network ls
what scenario you use host ?
- to use single container.
# docker run -itd -p --network=mynw nginx
# docker network inspect mynw
you will see different ip range.

# docker network
you will see output (help)
you can create, disconnect network.
by default bridge network.

# docker network create \
--driver=bridge
--
...

# docker run -itd -p 8080:8080 --name=jen1 jenkins/jenkind:lts
@docker hub,
search for jenkins

@the browser
IP:8080
# docker ps
# docker logs jen1
login to jenkins
ip:8080
user: docker
pw: ***
start using jenkins
create a job and run a sample job..
build now, and other, instlal plugins
lets asume, we just configure jenkins and everything is done..
go back to command prompt, and run,
# docker rm jen1
everything is gone
lets try again 
Lets try,
# docker run -itd -p 8080:8080 --name=jen1 jenkins/jenkind:lts
You lost all configuration information. 
we should be able to store the configuration. so how to make data safe and secure ..
store the data secure..
anything we can configure save it. 
- store your data in storage -> on vm or ebs - leastic block store or blog on azure..

@azure -> search, storage -> we will use it..
# docker run -itd -p 8080:8080 --name=jen1 jenkins/jenkind:lts
docker can not store data permanently
google "docker volume"
# docker volume ls
you see volumes but when you remove containers, these will be removed.
Lets work on volume
create a directory and store some content
# mkdir test; cd test
# cat > file1
from host machine
Now, mount it to container
# docker run -itd -v /root/test:/app --name=v1 ubuntu
-v -> volume
/app -> is a mount point to the container
# docker exec -it v1 bash
# cd /app; cat >file1
from container
# cat file1
you see the content
# exit
# cat file2
now, you can see it
# docker exec -it v1 bash
# exit
# docker rm -f v1
# ls 
you will see the data
-----------------------
lets try another eg
# mkdir jenk; chmod 777 jenks
# docker rm -f jen1
# docker run -itd -p 8080:8080 -v /root/jenk:/var/jenkins_home --user=jenkins --name=jen1 jenkins/jenkins:lts

/var/jenkins_home - this is the storage info for jenkins. all configures, plugins will be stored here.
# docker ps
if everything is create properly, 
# ls -ltr jenks
cd jenks/secrets
# cat *password*
get the pw and go to the browser
# http://ip:8080
login and all plugins will be loaded
create a siple job
item name: test
pipeline
hello world
build now
done..
go back to command line
# docker stop jen1
# docker rm jen1
# refresh the browser, everything is gone
docker run -itd -p 8080:8080 -v /root/jenk:/var/jenkins_home --user=jenkins --name=jen1 jenkins/jenkins:lts
login, you will be able to see everything.
your job is there..
As of now, we added physical volume.
How do use dynamically?
you can do dynamically.
# docker volume create --help
# docker volume create myvol
# docker volume inspect myvolume
review the output.
attach
# docker run -itd -v myvol:/app ubuntu
# docker ps
# docker exec -it <container id> bash
# df -h
# cd /apps; cd /apps; touch best
# exit
# cat best

create docker volume with elastic block storage
# docker volume create ebs....
download volume plugin for aws EBS.
https://docs.docker.com/storage/volumes

clean all containers
# docker ps -a -q
all container id will be listed
# docker rm -vf $(docker ps -a -q)
# docker ps -a -q
Everything is gone.
# docker images -q
# docker rmi -f $(docker ps -a -q)
All images are also deleted.

As of now,
job,
service
access
network
persist
next session:
how to build an image without manual effort.
docker file
docker-compose
docker swarm
@azure
go to resource group
and delete
@aws
go to ec2 instance and stop the instance

===================================