How to do Pagination in DynamoDB

In this article, we will discuss how to do pagination in DynamoDB.
LastEvaluatedKey
from the response as the value for ExclusiveStartKey
in the query parameters.Why do we need Pagination?
If we're using a normal relational database such as Oracle, or Postgres - you would be able to retrieve all the records in a table in a single query.
But, DynamoDB has a constraint - your response payload should not exceed 1 MB. This is to make sure that the performance is constant for even heavy workloads. So, how can we retrieve all the records, in case we want to do something with them?
Pagination in DynamoDB:
If there are more items for the current request, DynamoDB sends a property by name LastEvaluatedKey
in the response. If the value of this property is not undefined
- it means that the request has more items to it. You can use the value of this property to ExclusiveStartKey
a property of  QueryComandInput
params to fetch the next set of results. You can repeat this process until you get undefined
value for LastEvaluatedKey

Code Example
Let's assume we've 1000 records in a table. The table has UserId
as primary key (string) and orderDate
as secondary key.

Let's also assume that returning 10 records would consume 1 MB.
In the below code snippet, we're using TypeScript to paginate through the dynamodb records.
export async function queryItems(userId: string) {
try {
const items: any[] = [];
let lastEvaluatedKey: Record<string, AttributeValue> | undefined =
undefined;
const params: QueryCommandInput = {
TableName: TABLE_NAME,
KeyConditionExpression: "userId = :userId",
ExpressionAttributeValues: marshall({
":userId": userId,
}),
Limit: 100,
};
do {
const command = new QueryCommand(params);
const data = await ddbClient.send(command);
items.push(
...(data.Items?.map((item) => unmarshall(item)) as Record<
string,
any
>[])
);
lastEvaluatedKey = data.LastEvaluatedKey;
if (lastEvaluatedKey != undefined) {
params.ExclusiveStartKey = marshall({
userId: lastEvaluatedKey.userId.S || "",
orderDate: lastEvaluatedKey.orderDate.S || "",
});
}
} while (lastEvaluatedKey != undefined);
return items;
} catch (err) {
console.log("err in queryItems:", err);
}
}
I'm using a simple do-while loop to execute the query again and again till we get LastEvaluatedKey
is undefined.
One more thing to note here is that when we send value to ExclusiveStartKey
we can either construct the object manually as DynamoDb expects or you can use the marshall
utility function provided by dynamodb.
Please let me know your thoughts in the comments.