Amazon AWS

Message Queueing in Node.js with AWS SQS

Hello. This tutorial will understand AWS SQS and create a simple nodejs application to send and receive messages to/from the queue.

1. Introduction

AWS SQS represents the simple queue service in the Amazon ecosystem. This service:

  • Enables a user to store the messages (up to 256 KB in any format such as JSON, XML, etc) in the queue while waiting for another processor to process them
  • Separates the application components so that they can run independently and allowing asynchronously communication between the components
  • Two types of queues are available in the Amazon ecosystem i.e. Standard (default) and FIFO (First-in-first-out) queues
  • It is a pull-based and not push-based mechanism
  • Messages in a queue are kept from 1 minute to 14 days with a default retention period of 4 days
  • SQS offers that messages will be processed at least once

1.1 Setting up Node.js

To set up Node.js on windows you will need to download the installer from this link. Click on the installer (also include the NPM package manager) for your platform and run the installer to start with the Node.js setup wizard. Follow the wizard steps and click on Finish when it is done. If everything goes well you can navigate to the command prompt to verify if the installation was successful as shown in Fig. 1.

sqs queue - npm installation
Fig. 1: Verifying node and npm installation

2. Message Queueing in Node.js with AWS SQS

To set up the application, we will need to navigate to a path where our project will reside. For programming stuff, I am using Visual Studio Code as my preferred IDE. You’re free to choose the IDE of your choice.

2.1 Application pre-requisite(s)

To successfully implement this tutorial we will need –

  • AWS CLI user who should have an SQS full access policy attached to it so that it can successfully perform the queue related operations from the nodejs application
  • SQS standard queue in the region of your choice. I preferred to choose one in ap-south-1 region

2.2 Setting up the implementation

Let us write the different files which will be required for practical learning.

2.2.1 Setting up dependencies

Navigate to the project directory and run npm init -y to create a package.json file. This file holds the metadata relevant to the project and is used for managing the project dependencies, script, version, etc. Add the following code to the file wherein we will specify the required dependencies.

package.json

{
  "name": "node-awssqs-app",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [
    "aws",
    "sqs",
    "aws-sdk",
    "express",
    "node"
  ],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "aws-sdk": "^2.974.0",
    "body-parser": "^1.19.0",
    "express": "^4.17.1"
  }
}

To download the dependencies navigate to the directory path containing the file and use the npm install command. If everything goes well the dependencies will be loaded inside the node_modules folder and you are good to go with the further steps.

2.2.2 Setting up Environment configuration

In the root folder create a folder named config and add the following content to the env.js file. The file will contain the details of the CLI user, Region, and AWS SQS details and you are free to change as per your configuration

env.js

const env = {
  IAM: {
    ACCESS_KEY: "YOUR_AWS_ACCESS_KEY",
    SECRET: "YOUR_AWS_SECRET_KEY",
  },
  REGION: "YOUR_AWS_REGION",
  API_VERSION: "2012-11-05",
  QUEUE_ENDPOINT:
    "https://sqs.YOUR_AWS_REGION.amazonaws.com/YOUR_AWS_ACCOUNT_ID/YOUR_SQS_STANDARD_QUEUE",
};

module.exports = env;

2.2.3 Creating a controller

In the root folder add the following content to the index.js file. The file will consist of the following endpoints and the application will be running on the port number – 6001.

  • Get SQS status
  • Send a message to a queue
  • Receive a message from a queue
  • Purge the entire queue (i.e. delete all messages from a queue)
  • Delete a message from a queue

index.js

const env = require("./config/env");

const express = require("express");
const app = express();
app.use(express.json());

// Config AWS
const AWS = require("aws-sdk");
AWS.config.update({
  apiVersion: env.API_VERSION,
  accessKeyId: env.IAM.ACCESS_KEY,
  secretAccessKey: env.IAM.SECRET,
  region: env.REGION,
}); // for simplicity. In prod, use loadConfigFromFile, or env variables

const sqs = new AWS.SQS();

// Routes

// endpoint - get sqs status
// http://localhost:6001/status
app.get("/status", (req, res) => {
  res.status(200).json({
    status: "ok",
    data: sqs,
  });
});

// endpoint - send message to queue
// http://localhost:6001/send
app.post("/send", (req, res) => {
  let student = {
    id: req.body["id"],
    name: req.body["name"],
    email: req.body["email"],
    phone: req.body["phone"],
  };

  let params = {
    MessageBody: JSON.stringify(student),
    QueueUrl: env.QUEUE_ENDPOINT,
  };

  sqs.sendMessage(params, (err, data) => {
    if (err) {
      res.status(500).json({
        status: "500",
        err: err,
      });
    } else {
      res.status(202).json({
        status: "accepted",
        messageId: data.MessageId,
        data: "Message sent to queue",
      });
    }
  });
});

// endpoint - receive message from queue
// http://localhost:6001/receive
app.get("/receive", (req, res) => {
  let params = {
    QueueUrl: env.QUEUE_ENDPOINT,
    VisibilityTimeout: 600, // 10 min wait time for anyone else to process
  };

  sqs.receiveMessage(params, (err, data) => {
    if (err) {
      res.status(500).json({
        status: "500",
        err: err,
      });
    } else {
      if (!data.Messages) {
        res.status(200).json({
          status: "ok",
          messageId: null,
          data: "Nothing to process",
        });
      } else {
        res.status(200).json({
          status: "ok",
          messageId: data.Messages[0].MessageId,
          // receiptHandleId: data.Messages[0].ReceiptHandle,
          data: JSON.parse(data.Messages[0].Body),
        });
      }
    }
  });
});

// endpoint - purge the entire queue
// http://localhost:6001/purge
app.get("/purge", (req, res) => {
  let param = {
    QueueUrl: env.QUEUE_ENDPOINT,
  };

  sqs.purgeQueue(param, (err, data) => {
    if (err) {
      res.status(500).json({
        status: "500",
        err: err,
      });
    } else {
      res.status(200).json({
        status: "ok",
        data: "Queue purged",
      });
    }
  });
});

// endpoint - delete a message
// http://localhost:6001/delete
app.delete("/delete", (req, res) => {
  // todo - to be implemented.
  res.status(501).json({
    err: "Not implemented",
  });
});

// start server
const port = 6001;
app.listen(port, () => {
  console.log("SQS example app listening at http://localhost:%s", port);
});

3. Run the Application

To run the application navigate to the project directory and enter the following command as shown in Fig. 2. If everything goes well the application will be started successfully on port number 6001.

sqs queue - starting the app
Fig. 2: Starting the application

4. Demo

You are free to use postman or any other tool of your choice to make the HTTP request to the application endpoints.

Endpoints

// HTTP GET – Get sqs status
http://localhost:6001/status

// HTTP POST – Send message to queue
// Sample request – 
// {
// 	"id": "a7e7c5ae-cdf2-4c2a-919f-e5f3ec602c2f",
// 	"name": "Daniel",
// 	"email": "Atlas",
// 	"phone": "daniel.atlas@example.com"
// }
// HTTP GET – Receive message from queue
http://localhost:6001/receive

// HTTP GET – Purge the entire queue
http://localhost:6001/purge

// HTTP DELETE – Delete a message from queue
http://localhost:6001/delete

That is all for this tutorial and I hope the article served you with whatever you were looking for. Happy Learning and do not forget to share!

5. Summary

In this tutorial, we learned about SQS and how to create a simple express js application to communicate with AWS SQS. This application exposes different endpoints to perform different tasks as mentioned in the controller file. You can download the source code and the postman collection from the Downloads section.

6. Download the Project

This was a tutorial to implement a queue functionality in the node js application.

Download
You can download the full source code of this example here: Message Queueing in Node.js with AWS SQS

Yatin

An experience full-stack engineer well versed with Core Java, Spring/Springboot, MVC, Security, AOP, Frontend (Angular & React), and cloud technologies (such as AWS, GCP, Jenkins, Docker, K8).
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Back to top button