S3 - 101

1. S3 - Simple Storage service

2. Object storage - files,videos,photos,media etc. 2 types of storage - block & object.

3. 0-5 terrabytes file size, unlimited storage - amazon checks the storage availability in each region & will provision SANs as needed

4. Files are stored in buckets - buckets in folder in amazon terms

5. Cloudberry - provides cool explorer type apps to access s3

6. S3 is a universal namespace- must be unique globally

http://s3-eu(region)-west/amazonaws.com/acloudguru

7. When you upload a file to S3 ok code of 200 means it was successful

8. Data consistency model for s3 -

a) Read after write consistency for PUTS of new objects

b) Eventual consistency for Override PUTS & DELETES (takes time to propogate)

9. S3 is a Key-Value store- Lexicographic design - so alphabetical order of file name

Key - name of file

Value - data or sequence of bytes

version id-

metadata - data about data like the date etc

Subresources- doesnot exist on its own. exists under a object

a) access control lists - users/groups that can have access to the object. This access can be defined to a file, object or a bucket also

b) Torrent- S3 supports Torrent bit protocol

10. S3 specs

a) built for 99.99% availability - SLA

b)guaranteest 99.9-11*9's for durability of information stored - can support failure of 2 data centers concurently

c)Tiered storage available - like 30 days old file put in this tier, the newer ones in the other tier etc

d) Lifecycle management - tier management configuration settings

e)versioning

f)encryption

g)secure your data using access control lists & bucket policies

11. S3 Storage Tiers/Classes

a) S3 - 99.99 availability & 11*9s 99.9 durability

b) S3- IA Infrequently accessed - less priced than s3 but charged for retrievals

c)Reduced Redundancy storage- much cheaper. but durability is 99.99 only

d) Glacier - archival only, restoration is slow may take 3-5 hours. Cheapest

Refer S3 Tiered Storage specs

Refer S3 vs Glacier

12. S3 Charges -

Charged for

a)Storage

b)Requests

c)Storage management

d) transfer pricing- data coming into S3 is free but transferring around costs

e)Transfer acceleration

S3-BUCKET Lab

1. created a bucket Storage services - click on S3

2. Uploaded a file.

3. Added users to the bucket

4. Can set encryptions - Client side encryption, Server side encryption with amazon provided keys (SSE with S3), SS encryption with KMS(SSE with KMS), SS encryption with customer provided keys (SSE-c)

5. Security is through ACL - Access control lists & bucket policies

6. By default all buckets & objects in it are private.

For a bucket there are 3 tabs shown - Overview, Properties,Permissions & Management

similarly for a file also.

Added tags to the object itself.- the tag added to the bucket doesnt get passed on

S3 Versioning Lab

1. Once version is enabled it can only be suspended but not be deleted.

2. It writes/stores every change to a file as a separate object- even delete is stored

3. Provides additional level of security - MFA (multi factor authentication) to enable versioning and delete capability

4.

Cross Region Replication

- Enabling CRR only copies the future changes from the source to the destination bucket

- the existing contents must be copied with aws cli

aws configure - will ask for access key code, access value, region.

aws s3 ls - lists the bucket

aws s3 cp --recursive sourcebucket name destibucket name

- delete markers are replicated but deleting individual versions or delete markers are not replicated

-cross region replication is at a high level

- the two buckets must be in unique regions

Lifecycle management /rules -lab

Glacier is not available in singapore & southamerica - so create your buckets for this lab in some other region

AWS console->services->storage->s3->create bucket(no caps allowed in bucket name)- in properties tab enable versioning

on selecting the bucket u get the bucket screen with overview,properties,permissions & management tab

click on management tab to find the lifecycle tab

on click of lifecycle tab it lets you create a lifecycle rule and the pop up dialog appears that takes you through the lifecycle rule setting process

The rule can be set to a bucket or an individual file

1. current versions-

a) settings to transfer to infrequent access - IA - has to be a min of 30 days

b) settings to transfer to glacier archival- has to be in ia for a min of 30 days - so 60 days form creation is minimum here

2. after file becomes previous version

a) settings to transfer to ia-

b) settings to transfer to glacier archival - here no limit on days

c) settings to expire - when this is set - only a delete marker is added against the current version at the expiry date. If delete has to happen then it has to be combined with the permanent delete option

Once you create the rule the summary screen gives you the transitions& expirations .

Exam tips

- lifecycle rule configs can be done in conjunction with versioning

- can be applied on current versions / previous versions

- transition rules - 30 days for IA & 60 days for glacier archiving

-permanently delete

CDN Overview -

Content delivery network - is a system of distributed networks that deliver webpages and other webcontents to a user based on their geographic location, the origin of the content & the content delivery server.

edgelocation - location where content will be cached and is different from AWS region

origin - the origin of all files which is distributed by the cdn. eg a website in london,europe

distribution - a cdn which consists of a cluster of edge locations.

a) web distribution - for websites only

b) RTMP - for media streaming- for dobi files with rtmp protocol

Exam tips

Edge location is not read only - write/put object is allowed which gets written to the object in the origin server

TTL- time to live- objects in the cache are present untiil TTL expires

You can clear cached object , but will be charged

Clount front distribution - Lab

Services->network& content delivery

create distribution

Origin domain- prefilled with the bucket names

Origin path - user friendly domain name otherwise its a collection of letters & number random

Origin Description - it must be unique within a distribution

an object may have multiple origins

-cache behaviour settings

read only for the distribution to the bucket/file access

allow http methods- just read or put,options etc

allow only logged in users - apply security to restrict only logged in/signed in users-restrict signed in urls

use origin cache headers

configure min,max ttls

-distribution settings

use all or specific edge locations - price class

alternate domain name

ssl certificate - default or client ssl certrification

you can apply geo restriction also

Security & encryption

Security

-Bucket policies & access control lists

-access control lists can drill down to specific objects in a bucket also

-objects in a bucket are by default private

-access logs - gives log of all requests/access done to your bucket. This can also be set up to other bucket or account

Encryption

In transit -

When object is transfer into or out of s3

SSL/TSL- https

In Rest

Server Side encryption - SSE

- SSE- with amazon provided keys - sse-s3

-SSE with aws key management service -SSE-KMS- they provide envelope/management of your encryption key- also provide order trail for the keys

-SSE with customer provided keys - SSE-C

Client side encryption

Storage Gateway

- connects in-Premise IT environment with cloud storage to provide secure & scalable data transfer & storage to AWS cloud environment from in premise

Your data center ->asynchronously replicate -> AWS(S3 or glacier)

-SG(storage gateway) is available as a software for download in the form of a VM image. It supports VMware Elxi or microsoft hyper v. Once installed at your data center

and connected to your aws account through the activation process- the software gateway can be set up using aws console with options that work for you

4 different types of gateways

-File Gateway - uses NFS - Network file system - to store flat files to s3. All data is stored only in S3. Nothing onsite

-Volume Gateway - For block storage - takes point in time snapshots and stores in s3 using Amazon - EBS - Elastic block storage

Blocks/snapshotsa are stored in incrementals - so only the latest changes are stored

a) Stored volume- data asynchronously backed up using iSCSI block storage. All data is stored in premise and backed up in s3

b) Cached volume - all data stored in S3- only the most frequently accessed data is cached on site

Gateway virtual tape library -

Used for backup and uses popula backup applications but with virtual tape cartridge

Snowball

Petabyte device -its a physical data transfer device - to effectively transfer large amounts of data into and out of AWS without the high network cost

Snowball - 80Tbyte data, onboard storage capabilities

SnowballEdge - 100TB - on board storage & computing capabilities - basically data processing in premise with lamda functions

1256 TB is 1 petabyte

& 1256 petabyte is 1 exabyte

Snowmobile - exabyte scale data transfer- at a time you can transfer 100 PB with snow mobile

Snowball lab

Its under migration in services

- click create job to create a job for AWS to send you the snowball. keep pressing next to enter your address details etc

the workflow block diagram is shown- job is midway when the snowball has been delivered to you

- open flap right & left of the cuboid narrow side. - one side is the actual snowball kindle. the other end has the access to ethernet cable etc. the top has the power jack - which needs to be connected to the power cable.

log on to aws and download the snowball client & install in your pc. get credentials & download the manifest

in cli

power up the kindle

/snowball -i internet ip -m the manifest name -u credentials

/snowball cp filename bucket link(can get from your create job request also) will be of the pattern s3//bucketname

starts copying

once done power off & create a job for aws to pick it up

Transfer Acceleration

- instead of directly uploading to a bucket it lets you upload to a edge location which then uploads to your bucket faster. this service comes at a additional cost. when this is enabled , a unique endpoint url is given which has s3-acclerated as its domain. through this link you'll write to the edge location instead of directly to the bucket.

Static website hosting

http://pri-staticwebsite.s3-website-us-east-1.amazonaws.com/

create s3 bucket and in properties enable static website. give public access to read - only then the website will work. add index & error htmls to the static website properties. create the mentioned index & error files and upload to bucket. now go to static website and click on the endpoint url the website displays.

the url is always http://bucketname.s3-website-regionname.amazonaws.com

Next up

Previous

Section-4-AWS Object Storage & CDN-S3,Glacier & cloudFront