Class Notes
Session 39-AWS-Storage Gateway/FSx/Spot instances
On prem
- Our database server is running on our datacenter.
-
mySQL
- /var/lib/mysql
scale ---->
IOPS
- Throughput
- Parallel processing
WebServer
- /var/www/html
Storage
- object
- file
- block
# systemctl start httpd
# systemctl status httpd
# curl 127.0.0.1
s3 -> scale to PT or more ... almost unlimited ..
we will take a space from s3 to local system
- but we mount in such a way that data stored on local system, will be stored in s3.
filesystem
blockstorage
- We will use nfs share.
bucket storage converted into filestorage.
connect cloud storage to local storage (gateway)
On prem server
- Mount a directory (From aws)
NAS (nfs)/SMB
SAN (iSCSI)
s3
- bucket
local server <--- NAS/SAN <--------S3
Storage appliance
- Hardware
- virtual
1. Login to aws console
- search for storage gateway ( hybird storage)
2. - start
3. - select gateway type
a. file --> select
b. volume
c. tape
On our on prem,
we want to get NAS(nfs) storage. -> storage gateway (file.nas)
gateway is a virtual system which has os, cpu/mem
4. Seect host platform (5 options)
- vmware esxi
- microsfot hper-V
- Linux KVM
- Amazon EC2 --> Select this option
- Hardware appliance
--------------------------------------------------
We are going to use AWS EC2 instance to install Gateway.
- Since we don't have any other env, we will be using ec2.
Think EC2 instance as an on prem environment.
click on amazon ec2
review the instruction for ec2.
instance type -> m4.xlarge (select it) Note: its not free.
select instance type m4
next -> you need public IP.
next -> storage -> this instance works as a storage gateway.
add harddisk : 200GB -> type general purpose ssd(gp2)
never delete
next -> tag
name: SG_Virt_ec2_NAS
next -> allow all traffic (Security)
port 2049 allow /add
next next finish
our storage appliance is installed, now we have to configure it.
-----------------------------------------------
Get the IP of the ec2 instance.
Now, go back to Gateway page where you had select host platform
click next
service end point
- end point:
select public
next - connect to gateway
specify IP address ->
connect tto gateway
next specify gateway name from the drop down
select time, and other stuffs
and click on activate
Gateway health log group -> disable
save and continue
now, you have your own storage gateway on the cloud. Think this is running on prem.
this host provides storage gateway features..
as of now, we don't have any block or san storage to share
Go to your storage gateway
we create this gateway as file type.
it can only supports NAS.
click on create volume - it will fail. you have to create volume type
for that click on create gateway
now, open a new aws console and create a new bucket
bucket name: lwgatewaybucket
disable public access
and create bucket.
copy the bucket name and go back to gateway
click on file share
configure file share setting
fileshare name: lwgatewayshare
select gateway name
next
storage class :s3 standard
next -> summary - review
click create
now to back tp gateway -> you will see 1 file share
this storage gateway is configure as gateway server.
now, we will mount the s3 on local system.
click on file share -> status : available
go all the down on the button, you will see the command to mount. coy the command
now, you can go back to your local system abd login
run df -h and review the ouput
now, run the command you copied earlier.
# mount -t nfs -o nolck,hard 172.31.42.94:/lwstgaewayb1 /var/www/html
you have to change private ip to public
go to the gateway, copy the ip
# mount -t nfs -o nolck,hard 15.34.54.23:/lwstgaewayb1 /var/www/html
# df -hT
review the output
size you see is 8.oE -> 8 exxabyte size.
cat > /var/www/html/index.html
hello, welcome to clob
data is created and everything is sync to s3
for SAN
go to storage gateway
-> create a new gateway and select volume gateway
two options
*cached volume
*stored volumes
select cached volume (low latency) [use two disk on efor buffer and anoher for data] for speed and performance
next
select the host platform
-> use ec2 instance type
again new instance
M4 -x.large
next next -> harddisk ->
select two disk size 150GB
review the docs.
Tag: name: SG-Virt_SAN
next-> finish
we created an EBS disk. system creates snapshot of ebs disk which is stored on s3.
----------------------------------------
Review
lets say we have two systems and one system is providing storage to the client.
A ----iSCSI -------> B
server client
server a provides storage to B.
server is target and client is initiator
SG will be target and it is getting storage from s3.
client is our local system.
-----------------------------------------------------
now launch an EC2 instance -> default everything
tag: san: initiator
allow everything -> next -> next and finish
configure gateway
after selcecting host platform -> select end poing
degind gateway
specify the gateway name
sg_SAN
activate
wait for a while while activiting
click configure loggin
(you see note: you need 150GB for buffer)
no loging - >
-> finish
now, click on create cvolume on gateway page
gateway: sg_SAN
capacity: 10
volume content
iscisc target name: lwsttarget
click on create volume
chap authintication: skip
now, we created one harddisk
this harddisk, connect to client
go to client system (local vm, on prem)
here you will attach harddisk
# fdisk -l
list you all disk on the system.
how to use iscsi
1. discoverry name
2. login
click on the iscsi and you will see a instruction at the bottom of the page
just install the softare
iscsi initiator
/service iscsid status
/etc/init/d/
after software install, discover
iscsiadm ... you need IP as well
from client, we are discoverying ..
iscsiadm --mode discovery --type sendtargets --portal 172.31.38.164:3260
....
follow the instruction
# cd /dev/disk/by-path
you will see the disk here
# ls -l
to delete: go to action and delete
Practice this lab twice, imp
store data on the cloud
- encription/scalibility is managed by aws
------------xxxxxxxxxxxxxxxxxxx---------------------
New topic
FXs
- idea here is windows server, desktop, they have different services. dns, active directory, web server, distributed services
Win ->DFS(like NFS)
- optimized for bigger data
search for 'windows server dfs'
how does microsfot dfs works?
windows complete filesystem is managed by aws called FSX
HPC - high performance computing
search for linux hpc server
FSx also available on linux
linux cluster
- luster for linux system
go to aws console and search fsx
read doc
select FSx fow indows
- click next
specify name: myfsx1
ssd
specify storage capacity: 100GB
specify the throughput capacity: 2048 M/s
Note: its not free service)
go down to windows authentication
- aws managed microsoft ..
.leave as it is and click on next
for linux you select FSX for luster
choose through put speed ..
specify storage capacity: 2.4 tb
rest of the options are same.
No comments:
Post a Comment