Releasing Tremor v0.10!

Summary

This release is the last minor release before we start turning Tremor into a truly distributed event processing and data distribution engine. We focused on small things that improve usability and ironed out some rough edges here and there.

TL;DR

• We extended our type system and codecs by the binary type.
• Elasticsearch offramp now supports Linked Transport.
• String interpolation done right, now with #{} instead of just {}
• Release now includes prebuilt binaries, DEB packages and RPMs

New Release Artefacts

To make your life easier installing and Tremor more pleasant, we added new release artefacts. We've now got prebuilt binaries wrapped up in a tar.gz for amd64 linux, DEB, and RPM packages. And, lets not forget our well-known Docker Image.

We are going to explore more channels for release artefacts in the future to get you up and running with Tremor in no time.

In addition to those new release channels, we enabled thin-lto for all builds, which should improve compile-time while providing all the benefits of full link time optimization (LTO). We also link OpenSSL, now statically, to avoid incompatibilities with OpenSSL versions provided by the OS.

Binary Type and Binary Codec

Tremor is built for handling JSON-like structured data. It can effectively handle wire formats like JSON, YAML, msgpack and many more. We can represent them all with the same internal model of dynamic structured values. One blind spot up to now has been binary data. The reason for this was that most of the aforementioned formats do not support raw binary data (except msgpack). But times have changed.

With Tremor, you can now receive, assemble and send binary data. Receiving is done by configuring your onramp of choice with the binary codec. Imagine building a HTTP proxy with Tremor, that is not interested in the actual body payload but does its internal routing work only by looking at the headers. Previously, Tremor had to parse and deserialize the whole body payload. Now, it is able to conveniently pass those bytes through without ever touching them, and thus be even faster and more efficient for similar use cases.

Here is an example config for the needed onramps and offramps;

onramp:
  - id: http_input
    type: rest
    codec: binary
    linked: true
    config:
      host: 0.0.0.0
      port: 8080

offramp:
  - id: http_output1
    type: rest
    codec: binary
    linked: true
    config:
      endpoint: http://host01.example.com
      method: GET

And here is a very simple corresponding pipeline:

select event
from in
where $request.method == "GET" && array::contains($request.headers["content-type"], "application/json")
into out;

You are able to assemble complex binary events from structured data within tremor-script itself, like TCP packets:

let tcp_packet = <<
  event.src.port:16,  event.dst.port:16,
  event.seq:32,
  event.ack:32,
  event.offset:4, event.res:4, event.flags:8, event.win:16,
  event.checksum:16, event.urgent:16,
  event.data/binary
>>;

Elasticsearch Offramp as Linked Transport

We enhanced our elastic offramp to emit 1 event back to Tremor for each document indexed to Elasticsearch, be it successful or not. You will get all the Elasticsearch metadata like the _version and _id of the indexed document and also the whole payload to be indexed, which is especially useful in the face of indexing errors, e.g. from mismatched document schemas.

An elastic offramp configured like this:

offramp:
  - id: elastic
    type: elastic
    linked: true
    config:
      nodes:
        - elastic01:9200
        - elastic02:9200
        - elastic03:9200

will, for an event like this:

{"data": [1, 2, 3]}

send you back a response payload like this if everything went well via the offramps out port:

{
    "source": {
        "event_id": "1:0:0",
        "origin": "tremor-file://root/data.json"
    },
    "payload": {
        "data": [1, 2, 3]
    },
    "success": true
}

with the following document metadata in $elastic:

{
    "id": "TxQutncB0ovN9WdBcg2i",
    "index": "tremor_test",
    "doc_type": "_doc",
    "version": 1
}

And, in case indexing failed, you would get an event like this via the offramps err port:

{
   "source": {
        "event_id": "1:0:0",
        "origin": "tremor-file://root/data.json"
    },
    "payload": {
        "data": [1, 2, 3]
    },
    "success": false,
    "error": {
        "caused_by": {
            "reason": "Current token (VALUE_NUMBER_INT) not of boolean type\n at [Source: (byte[])\"POST //_bulk HTTP/1.1\r\ncontent-type: application/json\r\ncontent-length: 346\r\nuser-agent: reqwest/0.9.24\r\naccept: */*\r\naccept-encoding: gzip\r\nhost: 127.0.0.1:9200\r\n\r\n{\"index\":{\"_index\":\"tremor_test\",\"_type\":\"_doc\"}}\n{\"data\":\"[1, 2, 3]\"}\n\"[truncated 10 bytes]; line: 1, column: 13]",
            "type": "json_parse_exception"
        },
        "reason": "failed to parse field [data] of type [boolean] in document with id 'TxQutncB0ovN9WdBcg2i'. Preview of field's value: '1'",
        "type": "mapper_parsing_exception"}
}

Such an error message will also contain the same metadata behing the $elastic metadata key. Having detailed error data and the original payload at hand will enable users to handle success and error in new ways: Retries in case of errors, signalling upstream applications, try something completely different with that event payload.

String Interpolation Done Right

We changed the syntax for string interpolation to now use #{} for interpolating arbitrary expressions into a string literal, as we discovered some quirks in how it worked before (with {}). Creating JSON object strings {"key": "value"} or regex quantifiers [1-9][0-9]{2, 3} have been inconvenient to write and there were surprises together with using the string::format function.

Where we previously had written:

let json_string = """
{{ "key": "value"}}
""";
let regex = "[1-9][0-9]\{2, 3\}";

we can now express as:

let json_string = """
{ "key": "value"}
""";
let regex = "[1-9][0-9]{2, 3}";

which is much better e.g. for templating purposes.

Questions/Comments

On the Community Discord: https://chat.tremor.rs