Note from the future: the mixer was built when patchbay evaluation was the per-process bottleneck. Turns out patchbays are seriously fast, and the mixer’s own parse loop (every byte gets parsed twice, once at the mixer and once at the worker) tends to cap before the workers do. Sustained-load measurements have the mixer pinned at one core while workers sit at ~50% each. Deviating from the clean single-loop model was a mis-step....
Market data moves fast. Imagine your data provider gives you a stream of JSON frames that carry several fields per message, and every downstream subscriber re-implements the same demux, round, dedupe, alert logic. At scale, doing this n times causes read/write amplification and much wasted work, not to mention subtle bugs. Doing it once at the broker means that subscribers can use subject filtering to pick the slice they actually need and ignore the rest....
There’s now a public monoblok demo server you can use with any NATS client to try it out. It is at nats://monoblok.rtd.pub:4222, docs at demo.md. Grab the nats CLI, save the demo server as a context once, and select it so you don’t have to type the URL every time: nats context save monoblok-demo --server nats://monoblok.rtd.pub:4222 nats context select monoblok-demo Now nats pub and nats sub go straight to the public demo server....
Want to jump straight to the code? A car publishes its engine RPM ten times a second. A market data feed ticks on every quote. A cheap temperature sensor posts a fresh reading every two seconds, mostly identical to the last one. Data moves quickly, but most of it is noise: a publisher that doesn’t know what its subscribers care about, and a fleet of subscribers each writing the same defensive code to round, debounce, and ignore the boring readings....
Watching a feed, deriving a few values, and firing an alert when something looks off shouldn’t require a Kafka cluster, a JVM, and three days of ceremony. For a lot of problems, it doesn’t. NATS is a natural hub for streaming values: lightweight, subject-routed, no schema enforcement. The missing piece is somewhere to do the computation. It turns out Excel is a left-field but fairly compelling answer. It puts stream processing in the hands of analysts who already know how to use it, without asking them to learn a new framework or become developers overnight....
The Excel C SDK dates from the early 1990s. Memory management is manual, the type system is painful, and there’s almost no tooling. Despite all of this, it remains the only way to build add-ins that run truly in-process with Excel, supporting multi-threaded recalculation and the full breadth of what the host application can do. If you want the best possible performance, you need an XLL. But it’s a foot gun....
I have used NATS on various projects over the years. It provides a high performance transport layer than can be used to connect applications and services. If you haven’t already heard of it, this video is a fantastic primer. NATS supports websocket connections. nats.ws runs in browsers. This means we can directly use NATS as the backend for browser-based applications that need to display realtime streams of data. Why is this interesting?...
Some technologies I’ve enjoyed using this year: Go: still a lovely, simple to use and learn language. I took the time to understand the fairly new (?) support for generics. When I return to a Go codebase I feel right at home. The code is incredibly easy to read and the standard library is a joy. It remains my preference for writing APIs. Consul and Envoy: when you have a lot of services to securely link together, this is a great combination....
DynamoDB is often a perfect fit as the primary, operational system of record store for many types of application. It is fast, maintenance free and (if you use it well) economical. However it cannot perform aggregations or provide analytics on the data it holds. Reflecting the same data in another store like Apache Druid is commonplace. The below video demonstrates this idea in operation. The DynamoDB system of record is updated and Apache Druid is then used to perform aggregations on up to date values....
Composing systems out of smaller microservices has been commonplace for several years now. One trade off is increased complexity around development environments. Suppose the system you are working on consists of hundreds of services that all potentially make requests to one and other. If you are unlucky you might be faced with the task of spinning everything up locally or within a new cloud provider account. This is costly and often too much work....
Back when we all used SQL databases, it was common to paginate through large result sets by appending LIMIT offset, rows per page to a SELECT query. Depending on the schema, data volume and database engine, this was inefficient to varying degrees. On smaller result sets and with the right indexes, it was… posssibly OK. On larger result sets, the high page numbers would get progressively slower. Databases like DynamoDB prevent this inefficiency by handling pagination differently....
Here is one of my bad ideas that was nevertheless fun to think through. I am not suggesting you actually do this for anything serious. Really, I’m not. Serverless data technologies already exist. The idea I really like ClickHouse. Compared to the expanse of complex software in the big data space, it’s refreshing to run a single process. Although not without its foibles, it’s very fast and versatile. Running it on Cloud Run is likely a bad idea....
This project started when my son asked me for a replica train departures board for Christmas. I thought this was a great idea and this looks to be a really neat implementation, but I wanted us to have a go at building one ourselves. I promised that if we failed miserably I’d buy him one! After getting some off-the-shelf code up and running, I wanted to rethink the software. The resulting code is available on GitHub....
Update February 2019 — this post still gets a few views. Some of the content probably still makes sense, but Druid has moved on a lot.Managed services, SQL support, BI tools and so on. I’m still happily using it in production! The big data technology space is vibrant but crowded. There is a plethora of technologies, often appearing to be in competition with each other. There are several excellent SQL-on-Hadoop projects, for instance....
Back in my early teens when I started developing what could loosely be called software, I didn’t know what I was doing. If it compiled, ran and produced mostly the expected results, then the job was done. As a new programmer, I was immensely productive. Of course, problems came when it was time to fix bugs or extend the software. It was often easier to just start again than to try and understand the rat’s nest of poorly structured and unintelligible code....