data

To be fair, it’s more accurate to say that I wrote a data compiler. Let me explain myself after that unsettlingly clickbaity title… I’m currently contracted on a project that requires the ingestion and fast serving of public data. This data arrives in huge, unruly CSV and JSON files. The largest is a 30 GB JSON file. The files describe one-to-many relationships. One record carries many tags and many observations, each with its own set of attributes....

Update February 2019 — this post still gets a few views. Some of the content probably still makes sense, but Druid has moved on a lot.Managed services, SQL support, BI tools and so on. I’m still happily using it in production! The big data technology space is vibrant but crowded. There is a plethora of technologies, often appearing to be in competition with each other. There are several excellent SQL-on-Hadoop projects, for instance....