How to do Pagination in DynamoDB

In this article, we will discuss how to do pagination in DynamoDB.

Why do we need Pagination?

If we're using a normal relational database such as Oracle, or Postgres - you would be able to retrieve all the records in a table in a single query.

But, DynamoDB has a constraint - your response payload should not exceed 1 MB. This is to make sure that the performance is constant for even heavy workloads. So, how can we retrieve all the records, in case we want to do something with them?

Pagination in DynamoDB:

If there are more items for the current request, DynamoDB sends a property by name LastEvaluatedKey in the response. If the value of this property is not undefined - it means that the request has more items to it. You can use the value of this property to ExclusiveStartKey a property of  QueryComandInput params to fetch the next set of results. You can repeat this process until you get undefined value for LastEvaluatedKey

Code Example

Let's assume we've 1000 records in a table. The table has UserId as primary key (string) and orderDate as secondary key.

Let's also assume that returning 10 records would consume 1 MB.

In the below code snippet, we're using TypeScript to paginate through the dynamodb records.

export async function queryItems(userId: string) {
  try {
    const items: any[] = [];
    let lastEvaluatedKey: Record<string, AttributeValue> | undefined =
      undefined;
    const params: QueryCommandInput = {
      TableName: TABLE_NAME,
      KeyConditionExpression: "userId = :userId",
      ExpressionAttributeValues: marshall({
        ":userId": userId,
      }),
      Limit: 100,
    };

    do {
      const command = new QueryCommand(params);
      const data = await ddbClient.send(command);
      items.push(
        ...(data.Items?.map((item) => unmarshall(item)) as Record<
          string,
          any
        >[])
      );

      lastEvaluatedKey = data.LastEvaluatedKey;

      if (lastEvaluatedKey != undefined) {
        params.ExclusiveStartKey = marshall({
          userId: lastEvaluatedKey.userId.S || "",
          orderDate: lastEvaluatedKey.orderDate.S || "",
        });
      }
    } while (lastEvaluatedKey != undefined);

    return items;
  } catch (err) {
    console.log("err in queryItems:", err);
  }
}

I'm using a simple do-while loop to execute the query again and again till we get LastEvaluatedKey is undefined.

One more thing to note here is that when we send value to ExclusiveStartKey we can either construct the object manually as DynamoDb expects or you can use the marshall utility function provided by dynamodb.

Please let me know your thoughts in the comments.