Welcome readers, in this tutorial, we will make an introduction to AWS S3 and perform some operations using AWS CLI.
1. Introduction
AWS represents the Amazon Web Services and it provides different services ranging from IaaS (Infrastructure as a Service), PaaS (Platform as a Service), or SaaS (Packaged software as a Service).
- It offers flexibility by giving the users flexibility of what to do and what not
- Cost-effective solutions
- Scalability/Elasticity techniques to automatically scale up/down the application infrastructure when demand increases or decreases respectively
- Provides end-to-end security configuration and privacy to its customers
- A virtual infrastructure that offers complete privacy and isolation of operations
1.1 AWS Storage Service (AWS S3)
AWS Storage Service or simply known as AWS S3 is an online storage facility for the users. It cheap, easy to set up and the user only pays for what they utilize. It offers,
- To host static web-content and data or even the dynamic pages
- Data storage for analytics
- Backup and archival of data
- Disaster recovery solutions
1.1.1 S3 Bucket
Amazon S3 Bucket has two primary entities i.e. Object and Bucket, where objects are stored inside the buckets. It provides high availability and durability solutions by replicating the data of one bucket in multiple data centers. Each AWS account offers 100 buckets as free, however, this count can be increased on submitting a request to the support center. Amazon S3 features:
- Each object in a bucket is given a unique id
- S3 allows a developer to upload/delete or read an object via the REST API
- S3 offers two read-after-write and eventual consistency models to ensure that every change command committed to a system should be visible to all the participants
- Objects stored in a bucket never leave it’s location unless the user transfer it out
- Objects can be made private or public and rights can be granted to specific users
1.1.2 S3 Object
S3 Object(s) are the basic entities stored in the Amazon S3 bucket and is a simple key-value store. An object in S3 consists of the following:
- Key – Represents the name assigned to the object
- VersionId – Represents the key and version ID uniquely identifies an object. It is a string that S3 generates we add an object to the bucket
- Value – The content which we are storing in the bucket and can range up to 5 TB in size
- Metadata – Represents the name-value pairs with which we can store information regarding the object
- Access Control Information – Through this we control access to the objects stored in Amazon S3
1.1.3 Setting up IAM user for CLI operations
If someone needs to go through the process of creating an IAM user and attaching the S3 Full access policy, please watch this video.
2. Using Amazon S3 with the AWS CLI
Here is a systematic guide for implementing this tutorial but before going any further I’m assuming that readers are aware of the AWS basics.
2.1 Application Pre-requisite
To start with this tutorial, I’m hoping that readers at present have the AWS CLI running on this local environment and have created an IAM user to perform the CLI operations. If someone needs to work on the installation, please download the MSI installation from this link and following the instructions given on the page.
2.2 Configuring AWS CLI
Once the CLI is successfully installed, we need to configure the application to be able to connect with the AWS account. For configuring we can use the following command –
AWS Configure
1 | aws configure |
This command prompts the below information where –
- AWS access Id and Secret Key: These are two required ones to perform the operations via CLI (i.e. Acts as the account credentials and are generated through AWS Identity and Access Management (IAM))
- Default Region and Output Format: These two are the optional ones where default region signifies the region where the requests will be sent to and the default output format signifies how the results are formatted
2.3 AWS S3 Operations using AWS CLI
Now let us go through some AWS CLI operations for S3.
2.3.1 List all buckets
To list all buckets we use the ls command. Let us understand this with the help of an example.
List All Buckets
1 | aws s3 ls |
If everything goes well the following output will be shown on the terminal.
2.3.2 List the contents of a bucket
To list the contents of a bucket we use the ls command with a flavor. Let us understand this with the help of an example.
List Bucket Contents
1 2 3 4 | # Syntaxaws s3 ls s3://<bucket_name># Exampleaws s3 ls s3://s3-aws-training-bucket-2020 |
If everything goes well the following output will be shown on the terminal.
2.3.3 Make a new bucket
To make a new bucket we use the mb command. Let us understand this with the help of an example.
Create New Bucket
1 2 3 4 | # Syntaxaws s3 mb s3://<bucket_name># Exampleaws s3 mb s3://aws-cli-s3-training-bucket-2020 |
If everything goes well the following output will be shown on the terminal.
2.3.4 Sync object(s) from local to the bucket
To sync the object(s) we use the sync command. Let us understand this with the help of an example.
Sync Object(s) from Local to Bucket
1 2 3 4 | # Syntaxaws s3 sync <local_folder_path> s3://<bucket_name># Exampleaws s3 sync <some_path>\unsamples s3://aws-cli-s3-training-bucket-2020 |
If everything goes well the following output will be shown on the terminal.
Once done we can verify the same from the S3 console as well.
2.3.5 Remove an object from the bucket
To remove an object from the bucket we use the rm command. Let us understand this with the help of an example.
Remove an object from S3 Bucket
1 2 3 4 | # Syntaxaws s3 rm s3://<bucket_name>/<object_name># Exampleaws s3 rm s3://aws-cli-s3-training-bucket-2020/cr9qp77q_xi.jpg |
If everything goes well the following output will be shown on the terminal.
To perform recursive delete we can use the following command –
1 | aws s3 rm --recursive s3://<bucket_name> |
2.3.6 Remove an empty bucket
To remove an empty bucket we use the rb command. Let us understand this with the help of an example.
Remove an empty bucket
1 2 3 4 | # Syntaxaws s3 rb s3://<bucket_name># Exampleaws s3 rb s3://aws-cli-s3-training-bucket-2020 |
If everything goes well the following output will be shown on the terminal.
That is all for this tutorial and I hope the article served you whatever you were looking for. Happy Learning and do not forget to share!
3. Summary
In this section, we learned the following:
- Introduction to AWS S3
- How to work with S3 using CLI
Developers can download the sample commands from the Downloads section.
4. Download the Eclipse Project
This was an example of performing AWS S3 CLI operations.
You can download the full source code of this example here: Using Amazon S3 with the AWS CLI
