Summary
Tremor is an open source event processing and data distribution engine designed to interpose or intercept data in complex systems primarily to inject quality of service, correct data flows and to enrich or support complex traffic shaping, routing and load-balancing based on contextual, situational or content derived dynamic conditions.
Tremor has a powerful ETL language designed for flexibly pattern matching structured data such as JSON efficiently (SIMD-accelerated), and a declarative flow-based SQL language designed for rich JSON-like data to describe data processing and data flow graphs with declarative QoS constraints.
Tremor has been in large scale global production at Wayfair for 2 years now, and is gaining adoption in other organizations.
In this release, we return to Tremor’s roots and focus on quality of service by adding linked-transports to interpose RPC-like and synchronous-blocking protocols. We add circuit breakers to manage upstream and downstream byzantine events, and we provide mechanisms that offer shades of delivery guarantees or intrinsic transactions that, when combined with suitable upstream or downstream systems, can produce lossless data transfers for certain routes whilst remaining best-effort for routes that do not support lossless delivery or recovery in and of themselves with minimal loss.
Lastly, but most importantly, this is the first release of Tremor as a Linux Foundation/CNCF early-stage sandbox project. We would like to thank everyone at Wayfair in Boston and Berlin, the Linux Foundation and the CNCF who facilitated, helped, nurtured, transitioned DNS entries and worked with the legals to make this transition to fully open governance happen. A huge thanks to Chris and Amye at the Linux Foundation.
TL;DR
Read on for details after these headings:
- Refactored Concurrency Model
- Unified Command Line Interface
- Linked Transports
- Circuit Breakers & ‘Guaranteed’ Delivery
- Behavioural Changes
Stop here, unless you like details.
Refactored Concurrency Model
The maturity of concurrency and synchronization primitives within the rust programming language has transformed massively over the past 18 months. As these primitives have now stabilised and libraries based on these primitives are now stabilising, we have migrated the tremor runtime model to be fully internally asynchronous, non-blocking and task-driven, where possible.
There are still code paths within tremor, such as in kafka adapters that integrate with native libraries where a configurable number of threads can be configured and spun up- but we no longer have a thread per pipeline model.
This means that tremor application designers can now configure, deploy and run 1000s of pipelines in the same instance of tremor. In concert with the modularity theme in tremor v0.8, the quality of life/service theme of v0.9 further expands the flexibility available to tremor application designers and operators to design very high performance and highly efficient data processing and distribution solutions based on tremor.
Feel the awesome power of asynchronous runtimes
Unified Command Line Interface
Tremor has matured to the point that we have outgrown the legacy developer and operator tooling. In this release, we have unified, enhanced and extended developer and operator tooling to boldly go into a much improved quality of life for tremor users.
The new unified command line tool tremor can generate auto-completion for your shell (anything from zsh to powershell), generate documentation for your modules, run Tremor, debug the language intermediate forms, and has benchmark, integration, command-driven and unit testing frameworks built in.
Everything is syntax-highlighted and should conform to whichever theme you have running in your terminal. It’s like a universal tool - but more reliable because it's German but cool, because it's a Berliner(in the JFK, not cake sense...)
Linked Transports
Linked Transports allow bidirectional event flows in a request/response style to be built and routed through a tremor pipeline. This enables Tremor users to introspect, route, or even directly reply to incoming messages and intercept and manipulate outgoing responses. Tremor can thus be used as proxy, load-balancer or API-Gateway before or in between existing REST APIs or websocket servers.
Users can now support and implement their own domain specific REST APIs that can interact with the wider ecosystem of systems interconnected with Tremor- roll your own control plane and consolidate to a single plane of control- without deploying a separate solution or service.
(note: replies can only be delivered to feet in 0.9, we’re trying to extend that to other limbs in later releases).
Circuit breakers & “Guaranteed” delivery
The signal and contraflow control events within Tremor have been built upon in v0.9 to provide an event acknowledgement mechanism- this, in turn, provides the foundation for guaranteed delivery and circuit breakers, which enables finer-grained declarative quality of service mechanisms within tremor.
We can now configure Tremor applications or algorithms to pick up where they left off even if the tremor processes are stopped and restarted. Under certain conditions (based on configuration), we can recover entire end-to-end flows- such as the lossless distribution of messages across kafka sources and sinks regardless of which participant stops and restarts.
If you choose UDP, we cannot guarantee lossless delivery; but Tremor, even on a horrible quality network, should still minimise loss and recover gracefully- it won’t do so without message loss unless there’s some application-level handling of recovery.
Fully lossless transactional delivery is 10% the throughput of regular data distribution. It's as slow as Apache Kafka for now- but we will put effort into tuning performance here if the use case arises.
Circuit breakers enable us to detect downstream system failures beyond backpressure, and make informed decisions: to either discard, reroute or stop dataflow. This allows a more ingrained control over behaviour in non-optimal conditions compared to “always drop” earlier tremor versions implemented.
Delivery Guarantees describes a set of functionality that improves the delivery guarantees of a tremor system. What guarantees can be given depends heavily on the upstream and downstream systems, and the protocols used to communicate with them.
Open Source Community Changes
Tremor’s proposal to to adjoin the CNCF as an early stage standbox project was accepted by the CNCF TOC, and formal onboarding into the CNCF began and completed on time in September. This means that the tremor project’s copyright is now held by the Linux Foundation on behalf of the maintainers.
In practice, what does this mean? Day to day, there is no impact- currently all maintainers of the tremor project are sponsored by Wayfair. But the project now also has the support of the Linux Foundation and the CNCF and this, in turn, makes it easier for external organizations to adopt and contribute to it in support of their own business needs and open source governance strategies.
Tremor is visible on the CNCF Sandbox Project page and via the CNCF Landscape. The tremor project can now access development services from the CNCF and LF, ranging from artwork and website design through to visibility, marketing and promotion of the project at LF, CNCF and other conferences and events.
Behavioral Changes
REST offramp/sink:
- In an effort to streamline, the REST sink no longer adds newlines for batched events; this can now be done using the “lines” postprocessor.
put, as an option, is no longer supported; instead,methodis now used.- Only supports one
endpointinstead of multiple. The Round-robin operator with multiple offramps can be used as a much more capable replacement.
Heredocs:
- Leading indentation is no longer stripped.
- Heredocs now perform string interpolation, allowing it to be used for templating of textual content. In result, {and} needs to be escaped.
- The tremor-server, tremor-query, tremor-script, and tremor-tool binaries are now all unified in the self-name tremor binary. The binary also replaces the legacy benchmark and integration testing frameworks: the 3rd party coyote command, test harness, and introduces a new unit testing facility for tremor-script. It’s one command to rule them all.
- Circuit breakers are enabled and active by default- this is a significant behavioral change in sinks/offramps that completely disconnect; this will now stop the message flow on sources/onramps. Care should be taken in a production environment with non-resilient, non-robust and non-redundant downstream connectivity as this is now an operational concern that can benefit from a much finer-grained configuration surface.
The generic::backpressure operator has been renamed qos::backpressure. Using the old name will emit a warning in this release, but this backwards compatibility consideration will be addressed and the old name removed in the next release.
The operators runtime::tremor, grouper::bucket and script inside of trickle have their error output renamed err.
Linked transports introduce a source nature to sinks and a sink nature to sources. This feature is in preview mode and how to configure it is very likely to change as usability issues or misconceptions are ironed out. (v0.10)
Questions/Comments
On the our discord server.