Saturday, April 17, 2021

Day39- AWS - Storage Gateway, FSx, Luster

 Class Notes

Session 39-AWS-Storage Gateway/FSx/Spot instances


On prem
- Our database server is running on our datacenter.
-
mySQL
- /var/lib/mysql

scale ---->

IOPS
- Throughput
- Parallel processing


WebServer
- /var/www/html


Storage
- object
- file
- block


# systemctl start httpd
# systemctl status httpd

# curl 127.0.0.1

s3 -> scale to PT or more ... almost unlimited ..

we will take a space from s3 to local system
- but we mount in such a way that data stored on local system, will be stored in s3.

filesystem   
blockstorage

- We will use nfs share.

bucket storage converted into filestorage.

connect cloud storage to local storage (gateway)


On prem server
- Mount a directory (From aws)


NAS (nfs)/SMB
SAN (iSCSI)


s3
- bucket

local server <---  NAS/SAN <--------S3

 


 
Storage appliance
- Hardware
- virtual



1. Login to aws console
- search for storage gateway ( hybird storage)

2. - start
3. - select gateway type
a. file  --> select
b. volume
c. tape

On our on prem,
we want to get NAS(nfs) storage. -> storage gateway (file.nas)

gateway is a virtual system which has os, cpu/mem

4. Seect host platform (5 options)
- vmware esxi
- microsfot hper-V
- Linux KVM
- Amazon EC2  --> Select this option
- Hardware appliance
--------------------------------------------------
We are going to use AWS EC2 instance to install Gateway.
- Since we don't have any other env, we will be using ec2.
Think EC2 instance as an on prem environment.

click on amazon ec2

review the instruction for ec2.

instance type -> m4.xlarge (select it) Note: its not free.

select instance type m4

next -> you need public IP.
next -> storage -> this instance works as a storage gateway.

add harddisk : 200GB -> type general purpose ssd(gp2)
never delete

next -> tag
name: SG_Virt_ec2_NAS
next -> allow all traffic (Security)

port 2049 allow /add
next next finish

our storage appliance is installed, now we have to configure it.
-----------------------------------------------
Get the IP of the ec2 instance.

Now, go back to Gateway page where you had select host platform

click next


service end point
- end point:
select public
next - connect to gateway
specify IP address ->
connect tto gateway


next specify gateway name from the drop down
select time, and other stuffs
and click on activate



Gateway health log group -> disable

save and continue

now, you have your own storage gateway on the cloud. Think this is running on prem.

this host provides storage gateway features..

as of now, we don't have any block or san storage to share

Go to your storage gateway
we create this gateway as file type.
it can only supports NAS.

click on create volume - it will fail. you have to create volume type
for that click on create gateway


now, open a new aws console and create a new bucket
bucket name: lwgatewaybucket

disable public access
and create bucket.

copy the bucket name and go back to gateway

click on file share
configure file share setting

fileshare name: lwgatewayshare
select gateway name
next

storage class :s3 standard
next -> summary - review

click create

now to back tp gateway -> you will see 1 file share

this storage gateway is configure as gateway server.

now, we will mount the s3 on local system.

click on file share -> status : available

go all the down on the button, you will see the command to mount. coy the command

now, you can go back to your local system abd login

run df -h and review the ouput
now, run the command you copied earlier.

# mount -t nfs -o nolck,hard 172.31.42.94:/lwstgaewayb1 /var/www/html

you have to change private ip to public
go to the gateway, copy the ip
# mount -t nfs -o nolck,hard 15.34.54.23:/lwstgaewayb1 /var/www/html

# df -hT
review the output
size you see is 8.oE -> 8 exxabyte size.

cat > /var/www/html/index.html
hello, welcome to clob

data is created and everything is sync to s3

for SAN
go to storage gateway
-> create a new gateway and select volume gateway
 two options
*cached volume
*stored volumes

select cached volume (low latency) [use two disk on efor buffer and anoher for data] for speed and performance
next
select the host platform
-> use ec2 instance type

again new instance
M4 -x.large
next next -> harddisk ->

select two disk size 150GB

review the docs.

Tag: name: SG-Virt_SAN

next-> finish

we created an EBS disk. system creates snapshot of ebs disk which is stored on s3.


----------------------------------------
Review
lets say we have two systems and one system is providing storage to the client.

A ----iSCSI ------->    B
server        client

server a provides storage to B.
server is target and client is initiator

SG will be target and it is getting storage from s3.
client is our local system.

-----------------------------------------------------

now launch an EC2 instance -> default everything
tag: san: initiator

allow everything -> next -> next and finish



configure gateway

after selcecting host platform -> select end poing
degind gateway
specify the gateway name
sg_SAN

activate
wait for a while while activiting
click configure loggin

(you see note: you need 150GB for buffer)

no loging - >

-> finish
now, click on create cvolume on gateway page
gateway: sg_SAN
capacity: 10
volume content
iscisc target name: lwsttarget
click on create volume

chap authintication: skip

now, we created one harddisk

this harddisk, connect to client

go to client system (local vm, on prem)
here you will attach harddisk

# fdisk -l
list you all disk on the system.

how to use iscsi
1. discoverry name
2. login


click on the iscsi and you will see a instruction at the bottom of the page

just install the softare
iscsi initiator
/service iscsid status
/etc/init/d/
after software install, discover

iscsiadm ... you need IP as well

from client, we are discoverying ..

iscsiadm --mode discovery --type sendtargets --portal 172.31.38.164:3260


....
follow the instruction

# cd /dev/disk/by-path

you will see the disk here
# ls -l


to delete: go to action and delete

Practice this lab twice, imp

store data on the cloud
- encription/scalibility is managed by aws


------------xxxxxxxxxxxxxxxxxxx---------------------

New topic

FXs
- idea here is windows server, desktop, they have different services. dns, active directory, web server, distributed services

Win ->DFS(like NFS)
- optimized for bigger data

search for 'windows server dfs'

how does microsfot dfs works?

windows complete filesystem is managed by aws called FSX

HPC - high performance computing

search for linux hpc server

FSx also available on linux
linux cluster
- luster for linux system

go to aws console and search fsx
read doc
select FSx fow indows
- click next
specify name: myfsx1
ssd
specify storage capacity: 100GB
specify the throughput capacity: 2048 M/s

Note: its not free service)
go down to windows authentication
- aws managed microsoft ..

.leave as it is and click on next

for linux you select FSX for luster
choose through put speed ..

specify storage capacity: 2.4 tb

rest of the options are same.

No comments:

Post a Comment

Git branch show detached HEAD

  Git branch show detached HEAD 1. List your branch $ git branch * (HEAD detached at f219e03)   00 2. Run re-set hard $ git reset --hard 3. ...