DynamoDB pagination with page numbers in URLs

Back when we all used SQL databases, it was common to paginate through large result sets by appending LIMIT offset, rows per page to a SELECT query. Depending on the schema, data volume and database engine, this was inefficient to varying degrees. On smaller result sets and with the right indexes, it was… posssibly OK. On larger result sets, the high page numbers would get progressively slower. Databases like DynamoDB prevent this inefficiency by handling pagination differently....

October 27, 2021 · Alex Reid

Storage and retrieval of comment statistics in DynamoDB, using index overloading and sparse indexes

This series of posts demonstrates efficient filtering and pagination with DynamoDB. Part 1: Duplicating data with Lambda and DynamoDB streams to support filtering Part 2: Using global secondary indexes and parallel queries to reduce storage footprint and write less code Part 3: How to make pagination work when the output of multiple queries have been combined Part 4: Storage and retrieval of comment statistics using index overloading and sparse indexes...

November 25, 2020 · Alex Reid

DynamoDB pagination when multiple queries have been combined

This series of posts demonstrates efficient filtering and pagination with DynamoDB. Part 1: Duplicating data with Lambda and DynamoDB streams to support filtering Part 2: Using global secondary indexes and parallel queries to reduce storage footprint and write less code Part 3: How to make pagination work when the output of multiple queries have been combined Part 4: Storage and retrieval of comment statistics using index overloading and sparse indexes...

November 24, 2020 · Alex Reid

Filtering with GSIs and parallel queries in DynamoDB

This series of posts demonstrates efficient filtering and pagination with DynamoDB. Part 1: Duplicating data with Lambda and DynamoDB streams to support filtering Part 2: Using global secondary indexes and parallel queries to reduce storage footprint and write less code Part 3: How to make pagination work when the output of multiple queries have been combined Part 4: Storage and retrieval of comment statistics using index overloading and sparse indexes...

November 21, 2020 · Alex Reid

Filtering without using filters in DynamoDB

It never ceases to amaze me just how much is possible through the seemingly constrained model that DynamoDB gives us. It’s a fun puzzle to try to support access patterns beyond a simple key value lookup, or the retrieval of an ordered set of items. The NoSQL gods teach us to store data in a way that mirrors our application’s functionality. This is often achieved by duplicating data so that it appears in multiple predefined sets for inexpensive retrieval....

November 9, 2020 · Alex Reid

Driving an OLED display with a Raspberry Pi and AWS IoT

This project started when my son asked me for a replica train departures board for Christmas. I thought this was a great idea and this looks to be a really neat implementation, but I wanted us to have a go at building one ourselves. I promised that if we failed miserably I’d buy him one! After getting some off-the-shelf code up and running, I wanted to rethink the software. The resulting code is available on GitHub....

December 19, 2019 · Alex Reid