clickhouse

To be fair, it’s more accurate to say that I wrote a data compiler. Let me explain myself after that unsettlingly clickbaity title… I’m currently contracted on a project that requires the ingestion and fast serving of public data. This data arrives in huge, unruly CSV and JSON files. The largest is a 30 GB JSON file. The files describe one-to-many relationships. One record carries many tags and many observations, each with its own set of attributes....

Here is one of my bad ideas that was nevertheless fun to think through. I am not suggesting you actually do this for anything serious. Really, I’m not. Serverless data technologies already exist. The idea I really like ClickHouse. Compared to the expanse of complex software in the big data space, it’s refreshing to run a single process. Although not without its foibles, it’s very fast and versatile. Running it on Cloud Run is likely a bad idea....

Update February 2019 — this post still gets a few views. Some of the content probably still makes sense, but Druid has moved on a lot.Managed services, SQL support, BI tools and so on. I’m still happily using it in production! The big data technology space is vibrant but crowded. There is a plethora of technologies, often appearing to be in competition with each other. There are several excellent SQL-on-Hadoop projects, for instance....