Pagination in GraphQL: Efficiently Retrieve and Manipulate Data

Hello readers. In this tutorial, we will understand and implement the pagination in graphql.

1. Introduction

GraphQL is an API syntax that defines how to fetch data from one or more databases. It was developed by Facebook to optimize the RESTful api calls.

• It is a data query and manipulation language for API’s. It is faster, simple, and easier to implement
• It follows the client-driven architecture and gives a declarative way to fetch and update the data
• It has a graphical structure where objects are represented by nodes and edges represent the relationship between the nodes
• Provides high consistency across all platforms
• It doesn’t have any automatic caching system

1.1 Application components in GraphQL

In graphql, there are two types of application components.

1.1.1 Service-side components

The server-side component allows parsing the queries coming from the graphql client applications and consists of 3 components i.e. query, resolver, and schema. Apollo is the most popular graphql server.

1.1.2 Client-side components

The client-side components represent the client which is a code or a javascript library that makes the post request to the graphql server. It is of two types i.e.

• GraphiQL – Browser-based interface used for editing and testing graphql queries and mutations
• Apollo client – State management library for javascript that enables local and remote data management with graphql. Supports pagination, prefetching data, and connecting the data to the view layer

1.2 Introduction to Pagination in GraphQL

Pagination in GraphQL refers to the process of retrieving a large set of data in smaller, manageable chunks or pages. It is a crucial technique when dealing with queries that return a significant amount of data to improve performance and reduce resource consumption. By paginating the results, you can limit the amount of data transferred over the network and ensure efficient data retrieval. In GraphQL, pagination is typically achieved using the first and after arguments in a connection-based approach. Here’s a breakdown of the key components involved in pagination:

• Connection: A connection is a GraphQL object type that represents a collection of items. It encapsulates the paginated data and provides metadata about the collection, such as the total count, pageInfo, and edges.
• Edges: Edges represent individual items within a connection. Each edge contains a cursor and the corresponding node, which represents the actual data item.
• Cursor: The cursor is a unique identifier that represents a specific position within a connection. Cursors are opaque strings, often encoded with a specific format, and are used to specify the starting point for fetching the next page of data.
• PageInfo: PageInfo is an object type within a connection that holds information about the pagination state. It includes fields like hasNextPage, hasPreviousPage, startCursor, and endCursor. These fields allow clients to navigate through the pages and determine their current position.

Pagination in GraphQL allows for efficient retrieval and presentation of large datasets, providing flexibility and control to clients while minimizing unnecessary data transfer. By utilizing connections, edges, and cursors, you can build robust and scalable pagination systems in GraphQL.

1.2.1 Pagination Techniques in GraphQL

GraphQL does not have built-in pagination support like traditional REST APIs. However, there are various techniques you can use to implement pagination in GraphQL. Here are some commonly used approaches:

1.2.1.1 Limit-Offset Pagination

This technique involves using the first and offset arguments in your GraphQL query. The first argument specifies the number of items to fetch, and the offset argument indicates the starting index of the items to retrieve. By incrementing the offset value, you can navigate through different pages of data.

1.2.1.2 Cursor-Based Pagination

This approach uses cursor-based pagination, where a cursor represents a specific position in a list of items. Instead of relying on offsets, you use a cursor to indicate the item from which to start fetching the next page of data. The server generates the cursor based on some ordering criteria, such as a timestamp or a unique identifier.

1.2.1.3 Relay Connection Specification

The Relay framework defines a standardized pagination specification that many GraphQL implementations adhere to. It introduces the concepts of edges, nodes, and pageInfo to represent the paginated data and provide metadata about the pagination. The edges field contains the actual data with associated cursors and pageInfo contains information like whether there is a next/previous page and the total count of items.

1.2.1.4 Keyset Pagination

Keyset Pagination, also known as Range Pagination, is a pagination technique used in GraphQL to efficiently retrieve data based on a specified range or keyset. It relies on the properties of the data being sorted and the use of cursors to navigate through the dataset. In Keyset Pagination, each item in the dataset is associated with a unique key or set of keys. These keys are used to determine the order of the data. The client specifies a starting key or keys, and the server returns a page of results starting from that key. The client can then use the key of the last item on the page as the starting point for the next page, and so on.

Keyset Pagination offers several advantages over other pagination techniques:

• Performance: Keyset Pagination can be highly performant, especially when dealing with large datasets. It leverages indexes and the ordered nature of the data to efficiently retrieve the next page of results without the need for costly offset calculations.
• Stability: Keyset Pagination provides stable pagination. If new items are inserted into the dataset, it does not affect the previous pages or their order. This is in contrast to techniques like Limit-Offset Pagination, where inserting new items can shift the entire pagination window.
• Flexibility: Keyset Pagination allows for more flexible queries. Clients can specify complex criteria for filtering and sorting the data based on the keys or properties of the items.

Keyset Pagination is a powerful technique for efficiently paginating through sorted data in GraphQL, providing performance and flexibility in handling large datasets.

It’s important to note that the specific pagination techniques may vary depending on the GraphQL server implementation or any additional libraries or frameworks you use.

2. Implementing Pagination in GraphQL

2.1 Setting up Node.js

To set up Node.js on Windows you will need to download the installer from this link. Click on the installer (also include the NPM package manager) for your platform and run the installer to start with the Node.js setup wizard. Follow the wizard steps and click on Finish when it is done. If everything goes well you can navigate to the command prompt to verify if the installation was successful as shown in Fig. 1.

2.2 Understanding project structure

To set up the application, we will need to navigate to a path where our project will reside and I will be using Visual Studio Code as my preferred IDE. Let a take a quick peek at the project structure.

2.3 Setting up project dependencies

Navigate to the project directory and run npm init -y to create a package.json file. This file holds the metadata relevant to the project and is used for managing the project dependencies, script, version, etc. Replace the generated file with the code given below –

package.json

{
  "name": "graphql-pagination",
  "version": "1.0.0",
  "description": "an example to implement pagination in graphql",
  "main": "index.js",
  "scripts": {
    "dev": "nodemon index.js",
    "start": "node index.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [
    "graphql",
    "pagination",
    "nodejs"
  ],
  "author": "javacodegeek",
  "license": "open-source",
  "dependencies": {
    "express": "^4.18.2",
    "express-graphql": "^0.12.0",
    "graphql": "^16.7.1"
  },
  "devDependencies": {
    "nodemon": "^2.0.22"
  }
}

Once the file is replaced trigger the below npm command in the terminal window to download the different packages required for this tutorial.

Downloading dependencies

npm install

2.4 Setting up mock data

Create a file responsible to mock the database. The below file in the src/datasource directory is responsible to hold the users’ data. The primary focus of this tutorial is to understand the pagination and hence we skipped the real database part.

usersData.js

// src/datasource/usersData.js

const allUsers = [
  {"id":"64a77afe47453ffb797b0086","name":"Fox Travis","email":"foxtravis@rodeomad.com"},{"id":"64a77afebe2faa678a5b55d6","name":"Juarez Guy","email":"juarezguy@rodeomad.com"},{"id":"64a77afe24d7c170e49dcefb","name":"Shana Richard","email":"shanarichard@rodeomad.com"},{"id":"64a77afe7884134be5f3a35a","name":"Atkins Mcintyre","email":"atkinsmcintyre@rodeomad.com"},{"id":"64a77afe17c524993103ea1b","name":"Rachael Cantrell","email":"rachaelcantrell@rodeomad.com"},{"id":"64a77afe60888e5f6c5beeef","name":"Watts Knox","email":"wattsknox@rodeomad.com"},{"id":"64a77afe60a73acb5c86042f","name":"Carter Foster","email":"carterfoster@rodeomad.com"},{"id":"64a77afeafef5e16a0a63042","name":"Harris Carr","email":"harriscarr@rodeomad.com"},{"id":"64a77afe05134f99290353c1","name":"Bernadine Savage","email":"bernadinesavage@rodeomad.com"},{"id":"64a77afe4448abee044e4f65","name":"Carly Brock","email":"carlybrock@rodeomad.com"},{"id":"64a77afe7020f274040827bb","name":"Wyatt Moon","email":"wyattmoon@rodeomad.com"},{"id":"64a77afe97188570315aa1d8","name":"Kathrine Fletcher","email":"kathrinefletcher@rodeomad.com"},{"id":"64a77afe8188026cde1008e4","name":"Richard Fernandez","email":"richardfernandez@rodeomad.com"},{"id":"64a77afe8b4f4eae3ac650f6","name":"Valerie Gamble","email":"valeriegamble@rodeomad.com"},{"id":"64a77afe73920d7eaf320fec","name":"Katrina Bradley","email":"katrinabradley@rodeomad.com"},{"id":"64a77afe2fbf663df399db74","name":"Stacy Salazar","email":"stacysalazar@rodeomad.com"},{"id":"64a77afe22e0daa850c0ddd6","name":"Molly Chan","email":"mollychan@rodeomad.com"},{"id":"64a77afedb1be6db434313cf","name":"Bonnie Norman","email":"bonnienorman@rodeomad.com"},{"id":"64a77afedd951e9f7969845d","name":"Valentine Logan","email":"valentinelogan@rodeomad.com"},{"id":"64a77afef84cfb21e2d025ba","name":"Alisha Leblanc","email":"alishaleblanc@rodeomad.com"},{"id":"64a77afe567873a1ec61d560","name":"Alexis Randall","email":"alexisrandall@rodeomad.com"},{"id":"64a77afe8e93fa73a55b3e7c","name":"Irene Skinner","email":"ireneskinner@rodeomad.com"},{"id":"64a77afef1179e3fa63db660","name":"Galloway Mcguire","email":"gallowaymcguire@rodeomad.com"},{"id":"64a77afeeed0d41c643750bf","name":"Hendricks King","email":"hendricksking@rodeomad.com"},{"id":"64a77afeaaf579379f71ea93","name":"Ashley Peterson","email":"ashleypeterson@rodeomad.com"},{"id":"64a77afe3583353865f7033a","name":"Dianne Barry","email":"diannebarry@rodeomad.com"},{"id":"64a77afe07ac7e9ba3915bdf","name":"Bessie Pena","email":"bessiepena@rodeomad.com"},{"id":"64a77afe05f11234cd1ffa27","name":"Johnson Velez","email":"johnsonvelez@rodeomad.com"},{"id":"64a77afec4611db0f3dfd335","name":"Cynthia Talley","email":"cynthiatalley@rodeomad.com"},{"id":"64a77afee64d46b7171815ee","name":"Robbins Vincent","email":"robbinsvincent@rodeomad.com"}
];

module.exports = allUsers;

2.5 Setting up a resolver

Create a file in the src/resolvers directory responsible to interact with the database and address the incoming query from the client.

• The file userResolvers.js contains the resolver functions for handling user-related operations in a GraphQL API. Let’s go through the description of the file:
• The file starts with the import of the allUsers data source, which represents the collection of users. This data source can be fetched from a database or any other data storage.
• The getUsers function is defined to handle the pagination logic. It takes in the first and after arguments, which are used to determine the number of users to fetch and the starting point for pagination.
• Within the getUsers function, pagination logic is applied based on the provided first and after arguments. It slices the allUsers array to fetch the appropriate subset of users.
• The paginated users are then transformed into an array of edges, where each edge contains the user data and a cursor. The cursor is set as the user’s ID.
• The hasNextPage flag is determined by comparing the length of the allUsers array with the length of the current paginated users’ array, taking into account the presence of the after argument.
• The startCursor and endCursor are set to the IDs of the first and last users in the paginated result, respectively.
• Logging statements are added to display the paginated users and the page information in the console for debugging purposes.
• Finally, the resolver object userResolvers is defined, with the users field resolving to the getUsers function by passing the first and after arguments.
• The userResolvers object is exported to be used in the GraphQL schema or other parts of the application.

userResolvers.js

// src/resolvers/userResolvers.js

// Fetch users from your data source
const allUsers = require("../datasource/usersData");

const getUsers = (first, after) => {
  // Apply pagination logic based on 'first' and 'after' arguments
  // Return the paginated results

  // Apply pagination
  let users = allUsers.slice(); // Copy the array
  if (after) {
    const startIndex = users.findIndex((user) => user.id === after);
    users = users.slice(startIndex + 1);
  }
  if (first) {
    users = users.slice(0, first);
  }

  const edges = users.map((user) => ({
    node: user,
    cursor: user.id
  }));

  const hasNextPage = allUsers.length > users.length + (after ? 1 : 0);
  const endCursor = users.length > 0 ? users[users.length - 1].id : null;
  const startCursor = users.length > 0 ? users[0].id : null;

  console.log("Paginated users:", users); // Log the paginated users

  const pageInfo = {
    totalCount: allUsers.length,
    endCursor,
    hasNextPage,
    startCursor
  };

  console.log("Page info:", pageInfo); // Log the page info

  return {
    edges,
    pageInfo
  };
};

const userResolvers = {
  users: ({ first, after }) => getUsers(first, after)
};

module.exports = userResolvers;

2.6 Setting up type definition

Create a file in the src/schema directory responsible to represent the type definition required for the tutorial.

• The file begins with the import of the buildSchema function from the graphql module. This function is used to build the GraphQL schema.
• The schema variable is defined, which stores the result of invoking the buildSchema function with a template string.
• Inside the template string, the schema defines a single query field named users. It takes in the first and after arguments and resolves to the UserConnection type.
• The UserConnection type represents a connection of users and consists of two fields: edges and pageInfo. The edges field is an array of UserEdge objects, while the pageInfo field represents the metadata about the pagination.
• The UserEdge type represents an edge in the connection and contains two fields: node, which refers to the User type, and cursor, which is a string used as a cursor for pagination.
• The User type represents an individual user and includes fields such as id, name, and email.
• The PageInfo type represents the metadata about the pagination and includes fields such as totalCount, endCursor, hasNextPage, and startCursor.
• Finally, the schema variable is exported to be used in other parts of the application, such as the resolver functions.

userSchema.js

// src/schema/userSchema.js

const { buildSchema } = require("graphql");

const schema = buildSchema(`
  type Query {
    users(first: Int, after: String): UserConnection
  }

  type UserConnection {
    edges: [UserEdge]!
    pageInfo: PageInfo!
  }

  type UserEdge {
    node: User!
    cursor: String!
  }

  type User {
    id: ID!
    name: String!
    email: String!
  }

  type PageInfo {
    totalCount: Int!
    endCursor: String
    hasNextPage: Boolean!
    startCursor: String
  }
`);

module.exports = schema;

2.7 Creating the main file

Create a file in the src directory that acts as an entry point for the application.

• The file server.js is responsible for setting up the Express server and configuring the GraphQL endpoint. Let’s go through the description of the file:
• The file begins with the import of the necessary dependencies: express and graphqlHTTP from the express-graphql package. These dependencies are required to create the server and handle GraphQL requests.
• The schema variable is imported from the userSchema.js file. It represents the GraphQL schema that defines the available queries, types, and relationships.
• The userResolvers variable is imported from the userResolvers.js file. It contains the resolver functions that handle the logic for each query and mutation defined in the schema.
• An Express app is created using express() and stored in the app variable.
• The GraphQL endpoint is defined using the app.use() middleware function. It specifies the path /graphql as the endpoint URL and configures graphqlHTTP as the middleware function to handle incoming GraphQL requests.
• The graphqlHTTP middleware function is configured with the following options:
  • schema: It is set to the imported schema variable, representing the GraphQL schema.
  • rootValue: It is set to the imported userResolvers variable, providing the resolver functions for the defined queries and mutations.
  • graphiql: It is set to true to enable the GraphiQL interface, which provides a graphical interface for testing and exploring the GraphQL API.
• The server is started by calling the app.listen() function, specifying the port number (9444 in this case) and a callback function to log a message indicating that the server is running.

server.js

// src/server.js

const express = require("express");
const { graphqlHTTP } = require("express-graphql");
const schema = require("./schema/userSchema");
const userResolvers = require("./resolvers/userResolvers");

// Create the Express app
const app = express();

// Define the GraphQL endpoint
app.use(
  "/graphql",
  graphqlHTTP({
    schema: schema,
    rootValue: userResolvers,
    graphiql: true // Enable GraphiQL for testing
  })
);

// Start the server
const port = 9444;
app.listen(port, () => {
  console.log(`Server running at http://localhost:${port}/graphql`);
});

3. Run the Application

To run the application navigate to the project directory and enter the following command as shown below in the terminal. The application will be started successfully on port number 9444.

Run command

$ npm run start

Once the application is started successfully open the browser and hit the below endpoint to view the query explorer.

Application endpoint

http://localhost:9444/graphql

You can download the sample queries to understand the pagination in graphql from the Downloads section.

4. Best Practices for Pagination

When implementing pagination in GraphQL, here are some recommended best practices for pagination:

• Use Pagination Arguments: Use standardized pagination arguments like first, last, before, and after to define the pagination parameters in your GraphQL queries. This provides a clear and consistent interface for clients to request paginated data.
• Provide Default Values: Set appropriate default values for pagination arguments. This helps clients by providing a reasonable default behavior and avoids the need for clients to always specify pagination parameters explicitly.
• Use Cursor-Based Pagination: Prefer cursor-based pagination over offset-based pagination. Cursor-based pagination provides better performance and stability, especially when dealing with large datasets and when new items can be inserted into the dataset.
• Return Metadata: Include metadata in the response to provide additional information about the pagination. This can include fields like hasNextPage, hasPreviousPage, startCursor, endCursor, and totalCount. These metadata fields assist clients in navigating and understanding the available pages of data.
• Consistent Ordering: Maintain consistent ordering of data across pages. Ensure that the order of items remains the same as the client paginates through different pages. This avoids confusion and prevents items from appearing or disappearing when navigating between pages.
• Limit Result Sizes: Limit the maximum number of items that can be requested on a single page. This helps prevent performance issues and excessive data transfer.
• Caching and ETag Support: Implement caching mechanisms and utilize ETags (entity tags) to allow clients to cache paginated responses. This can significantly reduce server load and improve overall performance.
• Error Handling: Implement proper error handling for pagination. Return appropriate error responses when clients request invalid or out-of-range pages. Communicate error details to clients to assist with debugging and troubleshooting.
• Test and Monitor Performance
• Documentation

By following these best practices, you can ensure a robust and efficient pagination implementation in GraphQL, providing a great experience for clients interacting with your API.

5. Advanced Pagination Features in GraphQL

GraphQL also supports advanced pagination features that provide more fine-grained control and flexibility. Here are some advanced pagination features:

• Windowed Pagination: Windowed Pagination allows clients to request a specific window or range of items within a larger dataset. It enables clients to retrieve data from a specific starting point and fetch a fixed number of items before and after that point. This is useful when clients need to display a continuous subset of data, such as a sliding window or infinite scroll.
• Nested Pagination: Nested Pagination allows for paginating nested fields within a GraphQL query. It enables paginating on fields that have lists as their values, allowing clients to paginate through individual elements within those lists. This feature is helpful when dealing with complex data structures or nested relationships.
• Connection Resolvers: Connection resolvers provide a customizable way to resolve paginated fields in GraphQL. Instead of relying on the default pagination behavior provided by the GraphQL server, connection resolvers allow you to define custom pagination logic, including data fetching, sorting, filtering, and cursor generation. This feature gives you fine-grained control over the pagination process.
• Relay Global Object Identification: Relay Global Object Identification is a feature provided by the Relay framework for identifying and retrieving individual objects in a globally unique way. It assigns a globally unique ID to each object, allowing clients to reference and fetch specific objects using their IDs. This feature is particularly useful when paginating through a large dataset with complex relationships.
• Prefetching and Batch Fetching: Prefetching and batch fetching techniques optimize the retrieval of paginated data by allowing the server to prefetch and batch multiple requests together. By analyzing the client’s query patterns and predicting the required data, the server can optimize the data fetching process and reduce the number of round trips between the client and server.
• Custom Pagination Directives: GraphQL allows you to define custom directives that can be applied to fields to modify their behavior. You can create custom pagination directives to add additional pagination features, such as filtering, sorting, or custom cursor generation. These directives provide a way to extend the default pagination behavior to suit your specific requirements.

6. Conclusion

In conclusion, pagination is an essential aspect of designing and implementing GraphQL APIs to efficiently handle large datasets and provide a smooth experience for clients. While GraphQL does not have built-in pagination support, various techniques, and best practices can be employed to achieve pagination functionality.

The basic pagination techniques, such as limit-offset pagination and cursor-based pagination, allow clients to retrieve a specific number of items and navigate through pages of data. These techniques provide control over the size of the result set and enable efficient fetching of subsequent pages.

Moreover, advanced pagination features in GraphQL offer additional flexibility and optimization options. Windowed pagination enables clients to request a specific window or range of items within a dataset, catering to use cases like sliding windows or infinite scrolling. Nested pagination allows pagination through nested fields, which is beneficial for dealing with complex data structures and nested relationships.

Connection resolvers provide a customizable approach to resolving paginated fields, empowering developers to define custom pagination logic and achieve fine-grained control over the pagination process. Relay Global Object Identification offers a standardized mechanism for globally identifying and fetching specific objects, ensuring consistency in pagination across complex datasets.

Other advanced features, such as prefetching and batch fetching, optimize data retrieval by reducing round trips and improving performance. Custom pagination directives allow for extending and modifying the default pagination behavior of GraphQL by adding additional features like filtering, sorting, or custom cursor generation.

Implementing best practices for pagination in GraphQL, including using standardized arguments, providing default values, maintaining consistent ordering, returning metadata, handling errors, and thoroughly testing performance, ensures a robust and efficient pagination implementation.

By understanding and utilizing these pagination techniques and features, GraphQL APIs can deliver optimized, scalable, and responsive paginated data, enhancing the user experience and enabling efficient data consumption in client applications.

7. Download the Project

This was a tutorial to implement pagination in graphql.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

I agree to the Terms and Privacy Policy

Sign up

Thank you!

We will contact you soon.