data

rtd.pub is a platform for easily connecting any real-time data source to Microsoft Excel. You can write code in Go and Python, or simply configure pre-built open source connectors. This is a design and approach I’ve been mulling over for the past couple of months. I have now implemented it. This, and of course spreadsheets in general, are somewhat boring. However, they’re the original low code/RAD tool: in terms of bang for buck and the control they put into the hands of domain experts, nothing comes close....

In the previous series of posts, we built a data model capable of filtering and paginating product comments with DynamoDB. This post explores how we could solve the same problem with Cloud Bigtable. You might wonder why another technology is now being discussed. It is my belief that a lot of the thinking that goes into a data model design is somewhat portable, whether it be DynamoDB, Cloud Bigtable, Cassandra, HBase, or maybe even Redis....

Today I discovered a ridiculously easy way to run a Druid cluster on GCP: flick a switch when creating a Cloud Dataproc cluster. It’s even a recent version (0.17 at time of writing). Great, right? (Assuming you don’t mind using something labelled alpha by Google.) Customisation There is literally no documentation other than the page I stumbled across: Cloud Dataproc Druid Component. After running up a small cluster, I noticed some things were missing:...

Update February 2019 — this post still gets a few views. Some of the content probably still makes sense, but Druid has moved on a lot.Managed services, SQL support, BI tools and so on. I’m still happily using it in production! The big data technology space is vibrant but crowded. There is a plethora of technologies, often appearing to be in competition with each other. There are several excellent SQL-on-Hadoop projects, for instance....