Hello. This tutorial will understand AWS SQS and create a simple nodejs application to send and receive messages to/from the queue.
1. Introduction
AWS SQS represents the simple queue service in the Amazon ecosystem. This service:
- Enables a user to store the messages (up to 256 KB in any format such as JSON, XML, etc) in the queue while waiting for another processor to process them
- Separates the application components so that they can run independently and allowing asynchronously communication between the components
- Two types of queues are available in the Amazon ecosystem i.e. Standard (default) and FIFO (First-in-first-out) queues
- It is a pull-based and not push-based mechanism
- Messages in a queue are kept from 1 minute to 14 days with a default retention period of 4 days
- SQS offers that messages will be processed at least once
1.1 Setting up Node.js
To set up Node.js on windows you will need to download the installer from this link. Click on the installer (also include the NPM package manager) for your platform and run the installer to start with the Node.js setup wizard. Follow the wizard steps and click on Finish when it is done. If everything goes well you can navigate to the command prompt to verify if the installation was successful as shown in Fig. 1.
2. Message Queueing in Node.js with AWS SQS
To set up the application, we will need to navigate to a path where our project will reside. For programming stuff, I am using Visual Studio Code as my preferred IDE. You’re free to choose the IDE of your choice.
2.1 Application pre-requisite(s)
To successfully implement this tutorial we will need –
- AWS CLI user who should have an SQS full access policy attached to it so that it can successfully perform the queue related operations from the nodejs application
- SQS standard queue in the region of your choice. I preferred to choose one in
ap-south-1region
2.2 Setting up the implementation
Let us write the different files which will be required for practical learning.
2.2.1 Setting up dependencies
Navigate to the project directory and run npm init -y to create a package.json file. This file holds the metadata relevant to the project and is used for managing the project dependencies, script, version, etc. Add the following code to the file wherein we will specify the required dependencies.
package.json
{
"name": "node-awssqs-app",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [
"aws",
"sqs",
"aws-sdk",
"express",
"node"
],
"author": "",
"license": "ISC",
"dependencies": {
"aws-sdk": "^2.974.0",
"body-parser": "^1.19.0",
"express": "^4.17.1"
}
}
To download the dependencies navigate to the directory path containing the file and use the npm install command. If everything goes well the dependencies will be loaded inside the node_modules folder and you are good to go with the further steps.
2.2.2 Setting up Environment configuration
In the root folder create a folder named config and add the following content to the env.js file. The file will contain the details of the CLI user, Region, and AWS SQS details and you are free to change as per your configuration
env.js
const env = {
IAM: {
ACCESS_KEY: "YOUR_AWS_ACCESS_KEY",
SECRET: "YOUR_AWS_SECRET_KEY",
},
REGION: "YOUR_AWS_REGION",
API_VERSION: "2012-11-05",
QUEUE_ENDPOINT:
"https://sqs.YOUR_AWS_REGION.amazonaws.com/YOUR_AWS_ACCOUNT_ID/YOUR_SQS_STANDARD_QUEUE",
};
module.exports = env;
2.2.3 Creating a controller
In the root folder add the following content to the index.js file. The file will consist of the following endpoints and the application will be running on the port number – 6001.
- Get SQS status
- Send a message to a queue
- Receive a message from a queue
- Purge the entire queue (i.e. delete all messages from a queue)
- Delete a message from a queue
index.js
const env = require("./config/env");
const express = require("express");
const app = express();
app.use(express.json());
// Config AWS
const AWS = require("aws-sdk");
AWS.config.update({
apiVersion: env.API_VERSION,
accessKeyId: env.IAM.ACCESS_KEY,
secretAccessKey: env.IAM.SECRET,
region: env.REGION,
}); // for simplicity. In prod, use loadConfigFromFile, or env variables
const sqs = new AWS.SQS();
// Routes
// endpoint - get sqs status
// http://localhost:6001/status
app.get("/status", (req, res) => {
res.status(200).json({
status: "ok",
data: sqs,
});
});
// endpoint - send message to queue
// http://localhost:6001/send
app.post("/send", (req, res) => {
let student = {
id: req.body["id"],
name: req.body["name"],
email: req.body["email"],
phone: req.body["phone"],
};
let params = {
MessageBody: JSON.stringify(student),
QueueUrl: env.QUEUE_ENDPOINT,
};
sqs.sendMessage(params, (err, data) => {
if (err) {
res.status(500).json({
status: "500",
err: err,
});
} else {
res.status(202).json({
status: "accepted",
messageId: data.MessageId,
data: "Message sent to queue",
});
}
});
});
// endpoint - receive message from queue
// http://localhost:6001/receive
app.get("/receive", (req, res) => {
let params = {
QueueUrl: env.QUEUE_ENDPOINT,
VisibilityTimeout: 600, // 10 min wait time for anyone else to process
};
sqs.receiveMessage(params, (err, data) => {
if (err) {
res.status(500).json({
status: "500",
err: err,
});
} else {
if (!data.Messages) {
res.status(200).json({
status: "ok",
messageId: null,
data: "Nothing to process",
});
} else {
res.status(200).json({
status: "ok",
messageId: data.Messages[0].MessageId,
// receiptHandleId: data.Messages[0].ReceiptHandle,
data: JSON.parse(data.Messages[0].Body),
});
}
}
});
});
// endpoint - purge the entire queue
// http://localhost:6001/purge
app.get("/purge", (req, res) => {
let param = {
QueueUrl: env.QUEUE_ENDPOINT,
};
sqs.purgeQueue(param, (err, data) => {
if (err) {
res.status(500).json({
status: "500",
err: err,
});
} else {
res.status(200).json({
status: "ok",
data: "Queue purged",
});
}
});
});
// endpoint - delete a message
// http://localhost:6001/delete
app.delete("/delete", (req, res) => {
// todo - to be implemented.
res.status(501).json({
err: "Not implemented",
});
});
// start server
const port = 6001;
app.listen(port, () => {
console.log("SQS example app listening at http://localhost:%s", port);
});
3. Run the Application
To run the application navigate to the project directory and enter the following command as shown in Fig. 2. If everything goes well the application will be started successfully on port number 6001.
4. Demo
You are free to use postman or any other tool of your choice to make the HTTP request to the application endpoints.
Endpoints
// HTTP GET – Get sqs status
http://localhost:6001/status
// HTTP POST – Send message to queue
// Sample request –
// {
// "id": "a7e7c5ae-cdf2-4c2a-919f-e5f3ec602c2f",
// "name": "Daniel",
// "email": "Atlas",
// "phone": "daniel.atlas@example.com"
// }
// HTTP GET – Receive message from queue
http://localhost:6001/receive
// HTTP GET – Purge the entire queue
http://localhost:6001/purge
// HTTP DELETE – Delete a message from queue
http://localhost:6001/delete
That is all for this tutorial and I hope the article served you with whatever you were looking for. Happy Learning and do not forget to share!
5. Summary
In this tutorial, we learned about SQS and how to create a simple express js application to communicate with AWS SQS. This application exposes different endpoints to perform different tasks as mentioned in the controller file. You can download the source code and the postman collection from the Downloads section.
6. Download the Project
This was a tutorial to implement a queue functionality in the node js application.
You can download the full source code of this example here: Message Queueing in Node.js with AWS SQS
