Full Grammar
Rule ModuleFile
The ModuleFile rule defines a module in tremor.
A module is a unit of compilation.
rule ModuleFile ::=
ModuleBody
;
Represents the compiled form of a tremor source file.
Currently this can be troy deployment files, trickle query files, or tremor script files.
This is part of of tremor's compiler and runtime and not user modifiable.
Rule ModuleBody
The ModuleBody rule defines the structure of a valid module in tremor.
Modules begin with optional module comments.
Modules MUST define at least one statement, but may define many.
Statements are ; semi-colon delimited.
rule ModuleBody ::=
( ModComment ) ? ModuleStmts
;
The body of a compiled tremor source file has a module level comment ( lines beginning with 3 hashes ) and the statements that form the logic of that module.
This is part of of tremor's compiler and runtime and not user modifiable.
Rule ModComment
The ModComment rule specifies module comments in tremor.
Documentation comments for modules are optional.
A module documentation comment begins with a ### triple-hash and they are line delimited.
Muliple successive comments are coalesced together to form a complete comment.
The content of a module documentation comment is markdown syntax.
rule ModComment ::=
'<mod-comment>'
| ModComment '<mod-comment>'
;
Example
Module level comments are used throughout the tremor standard library and used as part of our document generation process.
Here is a modified snippet from the standard library to illustrate
### The tremor language standard library it provides the following modules:
###
### * [array](std/array.md) - functions to deal with arrays (`[]`)
### * [base64](std/base64.md) - functions for base64 en and decoding
### * [binary](std/base64.md) - functions to deal with binary data (`<< 1, 2, 3 >>`)
### * [float](std/float.md) - functions to deal with floating point numbers
### * [integer](std/integer.md) - functions to deal with integer numbers
### * [json](std/json.md) - functions to deal with JSON
...
Rule DocComment
The DocComment rule specifies documentation comments in tremor.
Documentation comments are optional.
A documentation comment begins with a ## double-hash and they are line delimited.
Muliple successive comments are coalesced together to form a complete comment.
The content of a documentation comment is markdown syntax.
rule DocComment ::=
'<doc-comment>'
| DocComment '<doc-comment>'
;
Example
Documentation level comments are used throughout the tremor standard library and used as part of our document generation process.
Here is a modified snippet from the standard library to illustrate
## Returns the instance of tremor.
##
## Returns a `string`
intrinsic fn instance() as system::instance;
...
This is a builtin function implemented in rust and used in a script as follows:
use tremor::system;
system::instance()
Rule ModuleStmts
The ModuleStmts rule defines a set of module statements.
Module statements are a ; semi-colon delimited set of ModuleStmt rules
rule ModuleStmts ::=
ModuleStmt ';' ModuleStmts
| ModuleStmt ';' ?
;
The set of computed statements in a compiled script.
This is part of of tremor's compiler and runtime and not user modifiable.
Rule ModuleStmt
The ModuleStmt rule defines the statement types that are valid in a tremor module.
rule ModuleStmt ::=
Use
| Const
| FnDefn
| Intrinsic
| DefineWindow
| DefineOperator
| DefineScript
| DefinePipeline
| DefineConnector
| DefineFlow
;
The set of statements that are legal in any tremor supported domain specific language.
This is part of of tremor's compiler and runtime and not user modifiable.
Rule ConfigDirectives
The ConfigDirectives rule allows line delimited compiler, interpreter or
runtime hints to be specified.
rule ConfigDirectives ::=
ConfigDirective ConfigDirectives
| ConfigDirective
;
See ConfigDirective for supported directives.
Rule ConfigDirective
A ConfigDirective is a directive to the tremor runtime.
Directives MUST begin on a new line with the #!config shebang config token.
rule ConfigDirective ::=
'#!config' WithExpr
;
Providing a metrics internal via a config directive
# Enable metrics with a 10 second interval
#!config metrics_interval_s = 10
Rule Use
Imports definitions from an external source for use in the current source file.
The contents of a source file form a module.
Multiple modules that share a common root can be combined into a single use clause by using use some::root::{m1, m2::sub_module::{m3, m4}}
TREMOR_PATH
The TREMOR_PATH environment path variable is a : delimited set of paths.
Each path is an absolute or relative path to a directory.
When using relative paths - these are relative to the working directory where the
tremor executable is executed from.
The tremor standard library MUST be added to the path to be accessible to scripts.
rule Use ::=
'use' UseTarget
;
Modules
Modules can be scripts. Scripts can store function and constant definitions.
Scripts are stored in .tremor files.
Modules can be queries. Queries can store window, pipeline, script and operator definitions.
Scripts are stored in .trickle files.
Modules can be deployments. Deployments can store connector, pipeline and flow definitions.
Deployments are stored in .troy files.
Conditioning
Modules in tremor are resolved via the TREMOR_PATH environment variable. The variable can
refer to multiple directory paths, each separated by a : colon. The relative directory
structure and base file name of the source file form the relative module path.
Constraints
It is not recommended to have overlapping or shared directories across the set of paths provided in the tremor path.
It is not recommended to have multiple definitions mapping to the same identifier.
Rule UseTarget
rule UseTarget ::=
MaybeAliased
| ModPath '::' MaybeAliased
| ModPath '::' '{' TargetList '}'
;
Rule TargetList
rule TargetList ::=
TargetElement
| TargetElement ',' TargetList
;
Rule TargetElement
rule TargetElement ::=
MaybeAliased
| ModPath '::' MaybeAliased
| ModPath '::' '{' TargetList '}'
;
Rule MaybeAliased
rule MaybeAliased ::=
Ident
| Ident 'as' Ident
;
Rule ArgsWithEnd
The ArgsWithEnd rule defines an arguments block with an end block.
rule ArgsWithEnd ::=
ArgsClause ? WithEndClause
|
;
An internal rule that defines an optional args block with and optional end token.
This rule is used and shared in other rules as part of their definitions.
Rule DefinitionArgs
The DefinitionArgs rule defines an arguments block without an end block.
rule DefinitionArgs ::=
ArgsClause ?
;
An optional argument block
args arg1, arg 2
This is a shared internal rule used in other rules as part of their definition.
Rule ArgsClause
The ArgsClause rule marks the beginning of an arguments block.
A valid clause has one or many argument expressions delimited by a ',' comma.
rule ArgsClause ::=
'args' ArgsExprs
;
args
x = y
Rule ArgsExprs
The ArgsExpr rule is a macro rule invocation based on the Sep separator macro rule.
An args expression is a comma delimited set of argument expressions.
rule ArgsExprs ::=
Sep!(ArgsExprs, ArgsExpr, ",")
;
Rule ArgsExpr
The ArgExpr rule specifies argument lists.
An argument can be an Ident or an assignment of the form <Ident> = <Expr>
rule ArgsExpr ::=
Ident '=' ExprImut
| Ident
;
As used in deployment rules to set or override arguments specifications. Arguments specifications define interface parameters that must be set by default ( or overridden ) for something to be well defined.
The tremor runtime checks for ommissions and produces an error for attempted instanciations that omit to provide a value for specified arguments.
Rule CreationWithEnd
The CreationWithEnd rule defines a with block of expressions with a terminal end keyword.
rule CreationWithEnd ::=
WithEndClause
|
;
with x = y end
Rule CreationWith
The CreationWith rule defines an optional with block of expressions without a terminal end keyword.
rule CreationWith ::=
WithClause
|
;
Rule WithClause
The WithClause rule defines a with block with a , comma delimited set of WithExpr rules.
rule WithClause ::=
'with' WithExprs
;
with x = y
Rule WithEndClause
The WithEndClause rule defines a with clause with an end terminal token.
rule WithEndClause ::=
WithClause 'end'
;
with x = y end
Rule WithExprs
The WithExprs rule defines a , comma delimited set of WithExpr rules.
rule WithExprs ::=
Sep!(WithExprs, WithExpr, ",")
;
Rule WithExpr
The WithExpr rule defines a name value binding.
rule WithExpr ::=
Ident '=' ExprImut
;
Form
name = <value>
Where:
nameis an identifier.<value>is any valid immutable expression.
Example
snot = "badger"
Rule ModularTarget
A ModularTarget indexes into tremor's module path.
In tremor a module is a file on the file system.
A module is also a unit of compilation.
A ModularTarget is a :: double-colon delimited set of identifiers.
Leading :: are not supported in a modular target..
Trailing :: are not supported in a modular target.
rule ModularTarget ::=
Ident
| ModPath '::' Ident
;
Examples
Loading and using a builtin function
# Load the base64 utilities
use std::base64;
# Base64 encode the current `event`.
base64::encode(event)
Loading and using a builtin function with an alias
# Load the base64 utilities
use std::base64 as snot;
# Base64 encode the current `event`.
snot::encode(event)
Rule Deploy
The Deploy rule defines the logical entry point into Tremor's command
oriented deployment syntax. The deployment grammar defines units of
deployment that the runtime manages on behalf of users.
The grammar embeds the statement oriented query syntax and expression oriented scripting syntax where appropriate.
A legal deployment is composed of:
- An optional set of module comments
- A sequence of top level expressions. There must be at least one defined.
- An optional end of stream token.
At least one deploy command.
rule Deploy ::=
ConfigDirectives ( ModComment ) ? DeployStmts
| ( ModComment ) ? DeployStmts
;
define flow test
flow
use std::time::nanos;
define connector metronome from metronome
with
config = {
"interval": nanos::from_seconds(1)
}
end;
define connector exit from exit;
define pipeline identity
args
snot = "badger",
pipeline
select args.snot from in into out;
end;
create connector metronome;
create connector exit;
create pipeline identity with
snot = "snot"
end;
connect /connector/metronome to /pipeline/identity;
connect /pipeline/identity to /connector/exit;
end;
deploy flow test
Rule DeployStmts
The DeployStmts rule defines the statements that are legal in a deployment
module.
Statements in a deployment modules are ; semi-colon delimited.
There MUST be at least one.
There MAY be more than one.
rule DeployStmts ::=
DeployStmt ';' DeployStmts
| DeployStmt ';' ?
;
Rule DeployStmt
The DeployStmt rule constrains the statements that are legal in a .troy deployment module.
Importing modules via the use clause is allowed.
Flow definitions and deploy commands are allowed.
rule DeployStmt ::=
DefineFlow
| DeployFlowStmt
| Use
;
Rule DeployFlowStmt
The DeployFlowStmt rule defines the content of a command to tremor
to deploy a flow.
The flows can be re-parameterized through overriding the default
parameters set in their originating source definitions via a with
clause.
rule DeployFlowStmt ::=
( DocComment ) ? 'deploy' 'flow' Ident 'from' ModularTarget CreationWithEnd
| ( DocComment ) ? 'deploy' 'flow' Ident CreationWithEnd
;
define flow test
flow
use std::time::nanos;
define connector metronome from metronome
with
config = {
"interval": nanos::from_millis(1)
}
end;
define connector exit from exit;
define pipeline identity
pipeline
select event from in into out;
end;
create connector metronome;
create connector exit;
create pipeline identity;
connect /connector/metronome to /pipeline/identity;
connect /pipeline/identity to /connector/exit;
end;
# The `deploy` statements commands tremor to instanciate the flow `test` - the flow in turn
# will result in a metronome, exit connector, pipeline to be started and interconnected as
# per the `test` definition above
deploy flow test
Rule ConnectorKind
The ConnectorKind rule identifies a builtin connector in tremor.
Connectors in tremor are provided by the runtime and builtin. They can be resolved through an identifier.
Examples
The http_server identifies a HTTP server connector.
The metronome identifies a periodic metronome.
rule ConnectorKind ::=
Ident
;
The name of a builtin connector implemented in rust provided by the tremor runtime.
Rule FlowStmts
The FlowStmts rule defines a mandatory ; semi-colon delimited sequence of FlowStmtInner rules.
rule FlowStmts ::=
FlowStmts_
;
Rule FlowStmts_
The FlowStmts_ rule defines a ; semi-colon delimited sequence of FlowStmtInner rules.
rule FlowStmts_ ::=
Sep!(FlowStmts_, FlowStmtInner, ";")
;
See FlowStmts rule for details.
This rule wraps away a lalrpop macro call for ease of reference in other rules in the grammar source.
Rule CreateKind
The CreateKind rule encapsulates the artefact types that can be created in the tremor deploymant language.
rule CreateKind ::=
'connector'
| 'pipeline'
;
An internal rule that allows connectors and pipelines to be created and used by the Create rule.
Rule FlowStmtInner
The FlowStmtInner rule defines the body of a flow definition.
rule FlowStmtInner ::=
Define
| Create
| Connect
| Use
;
- A pipeline or connector definition
- A pipeline or connector instance via
create - A
usestatement to import a definition - A
connectstatement to interlink instances
Rule Define
The Define rule allows connectors and pipelines to be specified.
rule Define ::=
DefinePipeline
| DefineConnector
;
Within a flow definition, allows pipeline and connectors to be defined.
Rule Create
The Create rule creates instances of connectors and pipelines in a flow.
rule Create ::=
'create' CreateKind Ident 'from' ModularTarget CreationWithEnd
| 'create' CreateKind Ident CreationWithEnd
;
Create a connector
create connector foo from snot::foo end;
Create a pipeline
create pipeline bar from badger::bar end;
Rule Connect
The Connect rule defines routes between connectors and pipelines running in a flow.
rule Connect ::=
'connect' '/' ConnectFromConnector 'to' '/' ConnectToPipeline
| 'connect' '/' ConnectFromPipeline 'to' '/' ConnectToConnector
| 'connect' '/' ConnectFromPipeline 'to' '/' ConnectToPipeline
;
Defines how to interconnect pipeline and connectors
Given
create connector ingest;
create connector egress;
create pipeline logic;
connect /connector/ingress/out to /pipeline/logic/in;
connect /pipeline/logic/out to /connector/egress/in;
Defines how the ingress, egress and logic runtime instances
are interconnected for data to flow through them in a specified
order.
Rule ConnectFromConnector
The ConnectFromConnector rule defines a route from a connector instance.
rule ConnectFromConnector ::=
'connector' '/' Ident MaybePort
;
connector/console/out
Connection from the console connector definition via the standard out port.
connector/console
The shorthand form where the standard out port is implied can also be used.
Rule ConnectFromPipeline
The ConnectFromPipeline rule defines route from a pipeline instance.
rule ConnectFromPipeline ::=
'pipeline' '/' Ident MaybePort
;
pipeline/filter/out
Connection from the filter pipeline definition via the standard out port.
pipeline/console
The shorthand form where the standard out port is implied can also be used.
Rule ConnectToPipeline
The ConnectToPipeline rule defines route to a pipeline instance.
rule ConnectToPipeline ::=
'pipeline' '/' Ident MaybePort
;
pipeline/filter/in
Connection to the filter pipeline definition via the standard in port.
pipeline/console
The shorthand form where the standard in port is implied can also be used.
Rule ConnectToConnector
The ConnectToConnector rule defines a route to a connector instance.
rule ConnectToConnector ::=
'connector' '/' Ident MaybePort
;
connector/console/in
Connection to the console connector definition via the standard in port.
connector/console
The shorthand form where the standard in port is implied can also be used.
Rule DefineConnector
The DefineConnector rule defines a connector.
A connector is a runtime artefact that allows tremor to connect to the outside world, or for the outside connector to connect to tremor to send and/or receive data.
The named connector can be parameterized and instanciated via the Create rule
rule DefineConnector ::=
( DocComment ) ? 'define' 'connector' Ident 'from' ConnectorKind ArgsWithEnd
;
use std::time::nanos;
define connector metronome from metronome
with
config = {
"interval": nanos::from_millis(1)
}
end;
Define user defind connector metronome from the builtin metronome connector
using a 1 second periodicity interval.
define connector exit from exit;
Define user dfeind connector exit from the builtin exit connector
with no arguments specified.
Rule DefineFlow
The DefineFlow rule defines a flow.
A flow is a runtime artefact that informs tremor how to interconnect and launch instances of pipelines and connectors.
A flow can define or use multiple in scope and already define pipelines and connectors and interconnect their streams together.
rule DefineFlow ::=
( DocComment ) ? 'define' 'flow' Ident DefinitionArgs 'flow' FlowStmts 'end'
;
define flow test
flow
use std::time::nanos;
define connector metronome from metronome
with
config = {
"interval": nanos::from_millis(1)
}
end;
define connector exit from exit;
define pipeline identity
args
snot = "badger",
pipeline
select args.snot from in into out;
end;
create connector metronome;
create connector exit;
create pipeline identity;
connect /connector/metronome to /pipeline/identity;
connect /pipeline/identity to /connector/exit;
end;
deploy flow test
A flow definition is a runnable or executable streaming program that describes the connectivity, the logic and how they are interconnected. A deploy statement is responsible for the actual deployment.
Rule Query
The Query rule defines the logical entry point into Tremor's statement
oriented query grammar. The grammar is embedded into deployments via
define pipeline statements.
Pipelines effectively provide a continous streaming abstraction for
event processing. The pipeline query is a graph of connected streams
and operators. The operators can be builtin or provided by users through
the script operator.
Window definitions for use in windowed statements such as those supported by
the select operation are also defined and named so that they can be used
by multiple operations and to compose tilt frames - chains of successive
windows of data in a streaming continuous select operation.
The syntax supports the definition of sub queries through the same define pipeline
syntax as in the deployment grammar.
rule Query ::=
ConfigDirectives Stmts
| Stmts
;
Rule Stmts
The Stmts rule defines a ; semi-colon delimited sequence of Stmt rules.
rule Stmts ::=
Stmt ';' Stmts
| Stmt ';' ?
;
Example for a query pipeline
use tremor;
create stream snot;
create stream badger;
# ...
Rule Stmt
The Stmt rule defines the legal statements in a query script.
Queries in tremor support:
- Defining named
window,operator,scriptandpipelinedefinitions. - Creating node instances of
stream,pipeline,operatorandscriptoperations. - Linking nodes togther to form an execution graph via the
selectoperation.
rule Stmt ::=
Use
| DefineWindow
| DefineOperator
| DefineScript
| DefinePipeline
| CreateOperator
| CreateScript
| CreatePipeline
| CreateStream
| OperatorSelect
;
Rule DefineWindow
The DefineWindow rule defines a temporal window specification.
A window is a mechanism that caches, stores or buffers events for processing over a finite temporal range. The time range can be based on the number of events, the wall clock or other defined parameters.
The named window can be instanciated via operations that support windows such
as the select operation.
rule DefineWindow ::=
( DocComment ) ? 'define' 'window' Ident 'from' WindowKind CreationWith ( ScriptState ) ? ( NamedScript ) ? ( EmbeddedScriptContent ) ? 'end'
;
define window four from tumbling
with
size = 4
end;
Rule DefineOperator
The DefineOperator rule defines an operator.
An operator is a query operation composed using the builtin operators provided by tremor written in the rust programming language.
The named operator can be parameterized and instanciated via the CreateOperator rule
rule DefineOperator ::=
( DocComment ) ? 'define' 'operator' Ident 'from' OperatorKind ArgsWithEnd
;
define pipeline subq
pipeline
define operator counter from generic::counter;
create operator counter;
select event from in into counter;
select event from counter into out;
end;
create pipeline subq;
select event from in into subq;
select event from subq into out;
Uses the builtin counter sequencing operator to numerate a stream.
Rule DefineScript
The DefineScript rule defines a named operator based on a tremor script.
A script operator is a query operation composed using the scripting language DSL rather than the builtin operators provided by tremor written in the rust programming language.
The named script can be parameterized and instanciated via the CreateScript rule
rule DefineScript ::=
( DocComment ) ? 'define' 'script' Ident DefinitionArgs ScriptState ? NamedScripts
;
define operator bucket from grouper::bucket;
define script categorize
script
let $rate = 1;
let $dimensions = event.logger_name;
let $class = "test";
event
end;
create operator bucket;
create script categorize;
select event from in into categorize;
select event from categorize into bucket;
select event from bucket into out;
Rule ScriptState
rule ScriptState ::=
'state' ExprImut
;
Rule NamedScripts
rule NamedScripts ::=
NamedScript NamedScripts
| EmbeddedScript
;
Rule NamedScript
rule NamedScript ::=
'script' 'from' Ident TopLevelExprs
;
Rule DefinePipeline
The DefinePipeline rule creates a named pipeline.
A pipeline is a query operation composed using the query langauge DSL instead of a builtin operation provided by tremor written in the rust programming language.
The named pipeline can be parameterized and instanciated via the CreatePipeline rule
rule DefinePipeline ::=
( DocComment ) ? 'define' 'pipeline' Ident ( 'from' Ports ) ? ( 'into' Ports ) ? DefinitionArgs Pipeline
;
define pipeline identity
pipeline
select event from in into out;
end;
Rule OperatorSelect
The OperatorSelect rule provides the select statement in streaming queries.
The statement has
- A target expression
- A stream and port from which to consume events
- A stream and port to which synthetic events are produced
- An optional set of iwndow definitions
- An optional
wherefilter - An optional
havingfilter - An optional
group by
Unlike ANSI-ISO SQL select operations in tremor do not presume tabular or columnar data. The target expression can be any well-formed and legal value supported by tremor.
rule OperatorSelect ::=
'select' ComplexExprImut 'from' StreamPort ( WindowClause ) ? ( WhereClause ) ? ( GroupByClause ) ? 'into' StreamPort ( HavingClause ) ?
;
The builtin select operator in queries.
A simple non-windowed non-grouped select
select event from in into out;
A simple non-windowed grouped select
select event from in into out group by event.key;
A windowed grouped select operation
select event from in[one_sec] by event.key into out;
Multiple windows can be configured in lower resolutions for multi-resolution windowed expressions where lower resolutions are merged into from higher resolution windows
select aggr::stats::hdr(event.count) form in[one_sec, fifteen_sec, one_min, one_hour] into out;
Rule CreateStream
The CreateStream allows users to create user defined streams beyond the basic
buitin set of in, out and err provided by the runtime for a pipeline query.
rule CreateStream ::=
'create' 'stream' Ident
;
create stream ctrl;
Creates a user defined stream by the provided name ctrl
Rule CreateScript
The CreateScript rule creates an operator based on a tremor script.
A script operator is a query operation composed using the scripting language DSL rather than the builtin operators provided by tremor written in the rust programming language.
The rule causes an instance of the referenced script definition to be created an inserted into the query processing execution graph.
rule CreateScript ::=
'create' 'script' Ident CreationWithEnd
| 'create' 'script' Ident 'from' ModularTarget CreationWithEnd
;
define operator bucket from grouper::bucket;
define script categorize
script
let $class = "test";
let $dimensions = [event.type, event.application];
let $rate = 1;
event;
end;
create operator bucket;
create script categorize;
select event from in into categorize;
select event from categorize into bucket;
select event from bucket into out;
Rule CreateOperator
The CreateOperator rule creates an operator.
An operator is a query operation composed using the builtin operators provided by tremor written in the rust programming language.
The rule causes an instance of the referenced operator definition to be created an inserted into the query processing execution graph.
rule CreateOperator ::=
'create' 'operator' Ident CreationWithEnd
| 'create' 'operator' Ident 'from' ModularTarget CreationWithEnd
;
# Define a round robin operator with 3 slots
define operator roundrobin from qos::roundrobin
with
outputs = ["one", "two", "three"]
end;
# create an instance of the operator
create operator roundrobin;
# Filter all inbound events into the rond robin
select event from in into roundrobin;
# Union slots inot outbound port
select event from roundrobin/one into out;
select event from roundrobin/two into out;
select event from roundrobin/three into out;
Rule CreatePipeline
The CreatePipeline rule creates a pipeline.
A pipeline is a query operation composed using the query langauge DSL instead of a builtin operation provided by tremor written in the rust programming language.
The rule causes an instance of the referenced pipeline definition to be created an inserted into the query processing execution graph.
rule CreatePipeline ::=
'create' 'pipeline' Ident CreationWithEnd
| 'create' 'pipeline' Ident 'from' ModularTarget CreationWithEnd
;
# Define a pipeline called `identity`
define pipeline identity
pipeline
select event from in into out;
end;
# Create an instance of the pipeline
create pipeline identity
Rule MaybePort
The MaybePort rule defines an optional Port.
rule MaybePort ::=
( '/' Ident ) ?
;
When interconnecting pipelines and connectors in flow definitions default ports can be inferred by the tremor runtime.
When an alternate port is required, the port specification can be used to explicitly select from available inbound or outbound ports.
Rule StreamPort
The StreamPort rule defines a stream by name with an optional named Port.
When the Port is omitted, tremor will internally default the Port to the
appropriate in or out port. Where the err or user defined Ports are
preferred, the optional Port specification SHOULD be provided.
rule StreamPort ::=
Ident MaybePort
;
in/snot
Within a query, allows the port of a specific stream to be referenced directly.
Rule WindowKind
Tumbling
A tumbling window defines a wall-clock-bound or data-bound window of non-overlapping
time for storing events. The windows can not overlap, and there are no gaps between
windows permissible.
Sliding
A sliding window defines a wall-clock-bound or data-bound window of events that captures
an intervalic window of events whose extent derives from the size of the window. A sliding
window of size 2 captures up to to events. Every subsequent event will evict the oldest and
retain the newest event with the previous ( now oldest ) event.
Conditioning
Both kinds of window store events in arrival order
rule WindowKind ::=
'sliding'
| 'tumbling'
;
Currently only tumbling is implemented.
Rule WindowClause
The WindowClause rule defines an optional window definition for a supporting operation.
rule WindowClause ::=
WindowDefn
;
Rule Windows
The Windows rule defines a sequence of window definitions that are , comma delimited.
rule Windows ::=
Windows_
;
Wraps the Windows_ internal rule for other rules to consume in their definitions
Rule Windows_
The Windows_ rule defines a sequence of window definitions that are , comma delimited.
rule Windows_ ::=
Sep!(Windows_, Window, ",")
;
A comma delimited set of window references.
Windows can be local or modular
win, my_module::one_sec, my_module::five_sec
The identifers refer to a window definition and can be used in operators to define a temporally bound set of events based on the semantics of the window definition.
In a tilt frame - or set of windows, the output of a window can is the input the next window in a sequence. This is a form of temporal window-driven event compaction that allows memory be conserved.
At 1000 events per second, a 1 minute window needs to store 60,000
events per group per second. But 60 1 second windows can be merged
with aggregate functions like dds and hdr histograms.
Say, each histogram is 1k of memory per group per frame - that is a cost of 2k bytes per group.
In a streaming system - indefinite aggregation of in memory events is always a tradeoff against available reosurces, and the relative business value.
Often multiple windows in a tilt frame can be more effective than a single very long lived window.
Rule Window
The Window rule defines a modular target to a window definition.
rule Window ::=
ModularTarget
;
Rule WindowDefn
The WindowDefn defines a temporal basis over which a stream of events is applicable.
rule WindowDefn ::=
'[' Windows ']'
;
Operations supporting windowed aggregate functions in tremor such as the select
statement can window incoming streams in the from clause:
select aggr::count(event) from in[one_second, ten_second]
...
Here, we stream events from the in stream into a one_second window.
Every second, we stream the aggregate result from the one second window
into the ten_second window.
So, even if we have 1 million events per second, the one_second and ten_second
windows will convert the event firehose into a trickle. Fun fact: this pun is where the
query language got its name from.
Rule WhereClause
The WhereClause defines a predicate expression used to filter ( forward or discard ) events in an operation.
The where clause is executed before a operation processes an event.
rule WhereClause ::=
'where' ComplexExprImut
;
select event from in
where present event.important
into out
The where filters events before computations occur upon them in operators
that support the clause. Any predicate ( boolean ) expression can be used
in a where filter.
Rule HavingClause
The HavingClause defines a predicate expression used to filter ( forward or discard ) events in an operation.
The having clause is executed after an operation has processed an event.
rule HavingClause ::=
'having' ComplexExprImut
;
select event from in
having present event.important
into out
The having filters events after computations has occured within them in operators
that support the clause. Any predicate ( boolean ) expression can be used
in a having filter.
When appropriate, the where clause should be used in preference over the having clause.
It is better to filter early before computation occurs when this is practicable or possible.
Rule GroupByClause
The GroupByClause defines the group by clause of a supporting operation in tremor.
An operator that uses a group by clause maintains the operation for each group captured by the grouping dimensions specified in this clause.
rule GroupByClause ::=
'group' 'by' GroupDef
;
select event from in[one_second]
having present event.important
group by event.priority
into out
The group by clause groups events based on a group expression. Each computed group effectively
has its own memory and computation allocated.
For windowed operations the windows are allocated for each group.
These groups and their windows are independant. This means that opening, closing, filling and recycling of windows is by group.
Rule GroupDef
The GroupDef rule defines the parts of a grouping dimension.
Group segments can be derived from:
- Expressions - for which their serialized values are used.
- Set expressions - which computes a set based on an expression.
- Each expressions - which iterates an expression to compute a set.
rule GroupDef ::=
ExprImut
| 'set' '(' GroupDefs ')'
| 'each' '(' ExprImut ')'
;
Example group definitions
A string value
event.string
The serialization of any legal tremor data value
event
A set based on multiple expressions:
set(event.key, state[key])
An set computed from an interation
let keys = ['snot', 'badger', 'goose'];
...
each(keys)
## Rule GroupDefs
The `GroupDefs` rule defines a `,` comma delimited set of `GroupDef` rules.
```ebnf
rule GroupDefs ::=
GroupDefs_
;
Wraps a macro call for use by other productions in the grammar
Rule GroupDefs_
The GroupDefs_ rule defines a , comma delimited set of GroupDef rules.
rule GroupDefs_ ::=
Sep!(GroupDefs_, GroupDef, ",")
;
A comma delimited set of GroupDef productions
Rule EmbeddedScriptImut
The EmbeddedScriptImut rule defines an optional embedded script.
rule EmbeddedScriptImut ::=
'script' EmbeddedScriptContentImut
;
Rule EmbeddedScriptContentImut
rule EmbeddedScriptContentImut ::=
ExprImut
;
Rule Ports
The Ports rule defines a , comma delimited set of stream ports.
rule Ports ::=
Sep!(Ports, <Ident>, ",")
;
A set of ports exposed in pipeline definitions in their from and into clauses
define pipeline example
from in, out, err, ctrl
into in, out, err, ctrl
pipeline
# A pipeline query implementation
...
end
The from and into ports do not need to be the same.
Tremor's compiler and runtime can use these definitions to validate deployments are correct, or discover deployments that are invalid. It is an error to send data to or receive data from a pipeline port that is not specified.
Rule OperatorKind
The OperatorKind rule defines a modular path like reference to a builtin tremor operator.
Operators are programmed in rust native code and referenced via a virtual module path.
rule OperatorKind ::=
Ident '::' Ident
;
A modular path identifying a builtin operator.
define operator roundrobin from qos::roundrobin;
Rule EmbeddedScript
The EmbeddedScript rule defines a script using the Script DSL [ Full ].
The script is enclosed in script .. end blocks.
rule EmbeddedScript ::=
EmbeddedScriptContent 'end'
;
script
event
end
Rule EmbeddedScriptContent
The EmbeddedScriptContent rule defines an embedded script expression.
rule EmbeddedScriptContent ::=
'script' TopLevelExprs
;
A single expression embedded within an embedded script
Rule Pipeline
The Pipeline rule defines a block of statements in a pipeline .. end block.
The block MAY begin with an optional set of ConfigDirectives.
rule Pipeline ::=
'pipeline' ConfigDirectives ? PipelineCreateInner 'end'
;
An internal rule to the DefinePipeline rule where the pipeline logic is provided.
Rule PipelineCreateInner
The PipelineCreateInner is an internal rule of the Pipeline rule.
The rule defines a ; semi-colon delimited set of one or many Stmts.
rule PipelineCreateInner ::=
Stmt ';' Stmts
| Stmt ';' ?
;
This rule allows queries to be defined in the context of a pipeline definition.
Rule Script
The Script rule defines the logical entry point into Tremor's expression
oriented scripting language. The scripting langauge can be embedded into
queries via the script operator. The scripting language is also used to
specify configuration of connectors, pipelines, flows, and operators in
the query language.
A legal script is composed of:
- An optional set of module comments
- A sequence of top level expressions. There must be at least one defined.
- An optional end of stream token
Scripting Language Entrypoint
This is the top level of the tremor scripting language tremor
rule Script ::=
( ModComment ) ? TopLevelExprs
;
Type system
Tremor supports a data oriented or value based type system with a syntax that is backwards compatible with JSON.
Any well-formed and legal JSON document is a valid literal in tremor.
Tremor literals for null, boolean, string ( utf-8 ), integer ( 64-bit unsigned ),
float ( 64-bit ieee ), arrays, and records are equivalent to their JSON counterparts.
Tremor also supports a binary literal for transporting and processing opaque binary data.
Asymmetric
JSON literals are valid tremor value literals.
Tremor literals MAY NOT always be valid JSON literal.
# The following literal is valid JSON and valid Tremor
[1, "snot", {}];
# The following literal is valid in tremor only
[1, "snot", {}, << data/binary >>, ];
Tremor supports comments, JSON does not. Tremor supports trailing commas in arrays and records, JSON does not. Tremor supports binary literal data, JSON does not.
Note: By default, most connectors in tremor serialize to and from json via a codec. The
type system in tremor however is agnostic to the wire format of data that flows through
tremor. So data originate as json, as msgpack.
Computations
Tremor also supports a rich expression language with the same support for additive, mutliplicate,
comparitive, and logical unary and binary expressions as languages like rust and java.
As most of the data that flows through tremor is heirarchically structured or JSON-like tremor also has rich primitives for structural pattern matching, structural comprehension or iterating over data structures.
Loops
Tremor does not support while loop or other primitives that can loop, recurse or iterate
indefinitely.
In an event based system, events are streaming continuously - so infinite loops that can block streams from making forward progress are considered harmful.
There are no loops.
We do support iteration over finite arrays.
We do support depth-limited tail recursive functional programming.
Expression oriented
The script processing is expression oriented. This is to say that every structural form supported by tremor returns a data structure as a result.
Event oriented
Scripts in tremor can emit or drop an `event that is being processed.
The event keyword is the subject. It identifies the value currently being processed.
The emit keyword halts processing succesfully with a value.
The drop keyword halts processing by discarding the current event.
Illustrative example
# Propagate events marked as important and convert them to system alerts
match event of
case %{ present important } => { "alert": event.message }
case _ => drop
end;
Rule TopLevelExprs
The TopLevelExprs rule defines semi-colon separated sequence of top level
tremor expressions with an optional terminating semi-colon
rule TopLevelExprs ::=
TopLevelExpr ';' TopLevelExprs
| TopLevelExpr ';' ?
;
The ToplEvelExprs specifies the expressions that are legal at the top level
of a script expression.
script
event.sum * 2
end
A sequence of ';' semi-colon delimited expressions of the following form are permissible:
- Constants
- Function definitions
- Intrinsic function definitions in the standard library
- Provided by the runtime to document builtins - not user modifiable without a pull request or feature enhancement
- Arbitrary complex expressions
- Use definitions
Rule InnerExprs
The InnerExprs rule defines the expression forms permissible within another
containing scope. Like TopLevelExprs, inner expressions are separated by semi-colons.
The semi-colon is optional for the last expression in a set of expressions.
At least one expression MUST be provided.
rule InnerExprs ::=
Expr ';' InnerExprs
| Expr ';' ?
;
Example
The following basic forms are permissible
for event.object of case (k, v) => v end;
for event.list of case (i, e) => e end;
match event of case %{} => "record" case _ => "not a record" end;
let list = event.list;
drop;
"any literal or basic expression";
1 + 2 * 3;
emit {"snot": "badger" }
# ...
Rule TopLevelExpr
The TopLevelExpr rule specifies the expression forms that are legal at the
outer most scope of a tremor script definition.
The legal forms are:
- Use declarations - these allow external modules to be referenced.
- Constant expressions - these are immutable compile time constants.
- Function definitions - these are user defined functions.
- Intrinsic function definitions - these are builtin funtions provided by the runtime.
rule TopLevelExpr ::=
Const
| FnDefn
| Intrinsic
| Expr
| Use
;
Example
In the tremor standard library many of the top level expressions
are use definitions importing sub modules from the module path.
use std::array; # Import the std array utilities
use std::base64 # Import the std base64 utilities;
use std::binary; # ...
use std::float;
use std::integer;
use std::json;
Rule Const
The Const rule defines a rule that binds an immutable expression to an identifier.
As the value cannot be changed at runtime.
Constants, by convention are ALL_UPPERCASE to distinguish them from variables. While it is allowed to deviate from this convention, it is not recommended and you will be yelled at by the compiler.
rule Const ::=
( DocComment ) ? 'const' Ident '=' ComplexExprImut
;
How do I create new immutable constant variable in tremor?
use std::base64;
const SNOT = "snot";
const BADGER = "badger";
const SNOT_BADGER = { "#{snot}": "#{base64::encode(badger)}" };
Rule Expr
The Expr rule aliases the SimpleExpr rule.
The alias allows higher levels of the DSL such as the rules in the deployment or query language to avoid some of the internal complexity in the scripting language.
Within the scripting DSLs grammar the different forms and variations of expression are significant.
However, in the higher level we limit exposure to a subset of these forms. This is done for convenience, and for consistency of usage, and ease of learning the language.
rule Expr ::=
SimpleExpr
;
The effective root of the subset of the expression langauge applicable in most processing context in tremor is captured by this rule.
Rule SimpleExpr
The SimpleExpr rule defines all the structural and simple expressions and literals in tremor.
rule SimpleExpr ::=
Match
| For
| Let
| Drop
| Emit
| ExprImut
;
Structural expressions
- Match
- For
Assignment
- Let
Flow
- Emit
- Drop
Basic expressions
Binary, unary and literal expressions
Rule AlwaysImutExpr
The AlwaysImutExpr defines the immutable expression forms in tremor.
Immutable expressions can be reduced at compile time and folded into literals.
rule AlwaysImutExpr ::=
Patch
| Merge
| Invoke
| Literal
| Path
| Record
| List
| StringLiteral
| BytesLiteral
| Recur
;
Rule Recur
The Recur rule defines stack-depth-limited tail-recursion in tremor functions.
rule Recur ::=
'recur' '(' ')'
| 'recur' '(' InvokeArgs ')'
;
fn fib_(a, b, n) of
case (a, b, n) when n > 0 => recur(b, a + b, n - 1)
case _ => a
end;
fn fib(n) with
fib_(0, 1, n)
end;
Tremor's functional programming langauge supports tail recursion via the
recur keyword. Tail recursion in tremor is limited to a fixed stack
depth - infinite recursion is not permissible.
Rule ExprImut
The ExprImut is the root of immutable expressions in tremor.
rule ExprImut ::=
OrExprImut
;
The effective root of the subset of the expression langauge applicable in most immutable processing context in tremor is captured by this rule.
Rule OrExprImut
The OrExprImut rule supports logical or expressions in tremor.
Binary logical or expressions take precedence over logical exclusive or expressions.
rule OrExprImut ::=
BooleanBinOp!(BinOr, ExprImut, XorExprImut)
| XorExprImut
;
let example = a or (b & 0);
Rule XorExprImut
The XorExprImut rule supports logical exclusive or expressions in tremor.
Binary logical exclusive or expressions take precedence over logical and expressions.
rule XorExprImut ::=
BooleanBinOp!(BinXor, XorExprImut, AndExprImut)
| AndExprImut
;
let example = a ^ (b & 0);
Rule AndExprImut
The AndExprImut rule supports logical and expressions in tremor.
Binary logical and expressions take precedence over bitwise or expressions.
rule AndExprImut ::=
BooleanBinOp!(BinAnd, AndExprImut, BitOrExprImut)
| BitOrExprImut
;
x + y
Rule BitOrExprImut
The BitOrExprImut rule supports bitwise or expressions in tremor.
Binary bitwise or expressions take precedence over bitwise exclusive or expressions.
rule BitOrExprImut ::=
BitXorExprImut
;
let example = a | (b & 0);
Rule BitXorExprImut
The BitXorExprImut rule supports bitwise exclusive or expressions in tremor.
Binary bitwise exclusive or expressions take precedence over bitwise and expressions.
rule BitXorExprImut ::=
BinOp!(BinBitXor, BitXorExprImut, BitAndExprImut)
| BitAndExprImut
;
let example = a ^ (b & 0);
Rule BitAndExprImut
The BitAndExprImut rule supports bitwise and expressions in tremor.
Binary bitwise and expressions take precedence over equality expressions.
rule BitAndExprImut ::=
BinOp!(BinBitAnd, BitAndExprImut, EqExprImut)
| EqExprImut
;
let example = a & (b & 0);
Rule EqExprImut
The EqExprImut rule supports equality expressions in tremor.
Binary equality expressions take precedence over comparitive expressions.
rule EqExprImut ::=
BinOp!(BinEq, EqExprImut, CmpExprImut)
| CmpExprImut
;
let example = a == (b & 0);
Rule CmpExprImut
The CmpExprImut rule supports comparative expressions in tremor.
Binary comparative expressions take precedence over bit shift expressions.
rule CmpExprImut ::=
BinOp!(BinCmp, CmpExprImut, BitShiftExprImut)
| BitShiftExprImut
;
x > y
Rule BitShiftExprImut
The BitShiftExprImut rule supports bit shift expressions in tremor.
Binary bit shift expressions take precedence over bitwise additive expressions.
rule BitShiftExprImut ::=
BinOp!(BinBitShift, BitShiftExprImut, AddExprImut)
| AddExprImut
;
let example = a >>> (b & 0);
Rule AddExprImut
The AddExprImut rule supports additive expressions in tremor.
Binary additive expressions take precedence over multiplicative expressions.
rule AddExprImut ::=
BinOp!(BinAdd, AddExprImut, MulExprImut)
| MulExprImut
;
x - y
Rule MulExprImut
The MulExprImut rule supports multiplicative expressions in tremor.
Binary multiplicative expressions take precedence over unary expressions.
rule MulExprImut ::=
BinOp!(BinMul, MulExprImut, UnaryExprImut)
| UnaryExprImut
;
x * y
Rule UnaryExprImut
The UnaryExprImut rule specifies unary expression operations.
Expressions can be marked as + positive, - negative explicitly when needed.
Otherwise, the expression reduces to a simple unary expression.
The simple unary expression has lower precedence.
rule UnaryExprImut ::=
'+' UnaryExprImut
| '-' UnaryExprImut
| UnarySimpleExprImut
;
Is the root expression for unary expressions in the language.
-(1)
Rule UnarySimpleExprImut
The UnarySimpleExprImut rule specifies predicate unary expression operations.
Expressions can be marked explicitly with not or ! to negate the target simple presence expression.
Otherwise, the expression reduces to a simple presence expression.
The simple presence expression has lower precedence.
rule UnarySimpleExprImut ::=
'not' UnarySimpleExprImut
| '!' UnarySimpleExprImut
| PresenceSimplExprImut
;
not (a and b)
Rule PresenceSimplExprImut
The PresenceSimplExprImut rule specifies presence and simple expressions
Expressions path predicate tests based on the present and absent predicate test
expressions, or a simple expression.
Otherwise, the expression reduces to a simple expression.
The simple expression has lower precedence.
rule PresenceSimplExprImut ::=
'present' Path
| 'absent' Path
| SimpleExprImut
;
present $.request.method
absent state.cache
Rule ComplexExprImut
The ComplexExprImut rule defines complex immutable expression in tremor.
rule ComplexExprImut ::=
MatchImut
| ForImut
| ExprImut
;
Rule Intrinsic
The intrinsic rule defines intrinsic function signatures.
This rule allows tremor maintainers to document the builtin functions implemented as native rust code. The facility also allows document generation tools to document builtin intrinsic functions in the same way as user defined functions.
In short, these can be thought of as runtime provided.
For information on how to define user defined functions see the function rule.
rule Intrinsic ::=
( DocComment ) ? 'intrinsic' 'fn' Ident '(' ')' 'as' ModularTarget
| ( DocComment ) ? 'intrinsic' 'fn' Ident '(' FnArgs ')' 'as' ModularTarget
| ( DocComment ) ? 'intrinsic' 'fn' Ident '(' FnArgs ',' '.' '.' '.' ')' 'as' ModularTarget
| ( DocComment ) ? 'intrinsic' 'fn' Ident '(' '.' '.' '.' ')' 'as' ModularTarget
;
Example
From our standard library generated documentation, we can see that the base64 encode function is an intrinsic function.
## Encodes a `binary` as a base64 encoded string
##
## Returns a `string`
intrinsic fn encode(input) as base64::encode;
Rule FnDefn
The FnDefn rule allows user defined functions to be defined.
This rule allows tremor users to create functions for reuse in one or many tremor applications.
rule FnDefn ::=
( DocComment ) ? 'fn' Ident '(' '.' '.' '.' ')' 'with' InnerExprs 'end'
| ( DocComment ) ? 'fn' Ident '(' FnArgs ',' '.' '.' '.' ')' 'with' InnerExprs 'end'
| ( DocComment ) ? 'fn' Ident '(' ')' 'with' InnerExprs 'end'
| ( DocComment ) ? 'fn' Ident '(' FnArgs ')' 'with' InnerExprs 'end'
| ( DocComment ) ? 'fn' Ident '(' ')' 'of' FnCases 'end'
| ( DocComment ) ? 'fn' Ident '(' FnArgs ')' 'of' FnCases 'end'
;
Pattern match based function arguments
Functions defined with an of keyword in their signature use pattern matching against arguments
fn fib_(a, b, n) of
case (a, b, n) when n > 0 => recur(b, a + b, n - 1)
case _ => a
end;
Ordinary functions
Functions defined with a with keyword in their signature use ordinary
arity based matching.
fn fib(n) with
fib_(0, 1, n)
end;
Function documentation
In modular functions, it is customary to provide user level documentation for the intended users of a function. Here is an example from the tremor standard library
### Trace Identifiers
###
###
use std::type;
use std::binary;
use std::array;
use std::string;
## Is the `trace_id` valid
##
## Checks the `trace_id` argument to see if it is a valid
## trace id. A legal trace id is one of:
##
## * An array of integers in the range of [0..=255] of length 8
## * A binary 16 byte value
## * A 32-byte hex-encoded string
## * An array of 16 int values
## * Regardless of representation, the value must not be all zeroes
##
## Returns a record when the representation is well-formed of the form:
##
## ```tremor
## {
## "kind": "string"|"binary"|"array", # Depends on input
## "valid": true|false, # True if well-formed and valid
## "value": "<trace_id>" # Representation depends on `kind`
## }
## ```
##
## Returns an empty record `{}` when the representation not well-formed
##
fn is_valid(trace_id) of
# String representation
case(trace_id) when type::is_string(trace_id) =>
{ "kind": "string", "valid": trace_id != "00000000000000000000000000000000" and string::bytes(trace_id) == 32, "value": trace_id }
# Binary representation
case(trace_id) when type::is_binary(trace_id) =>
let arr = binary::into_bytes(trace_id);
{ "kind": "binary", "valid": binary::len(arr) == 16 and trace_id != << 0:64, 0:64 >>, "value": trace_id }
# Array representation
case(trace_id) when type::is_array(trace_id) =>
{ "kind": "array", "valid": array::len(arr) == 16 and trace_id != [ 0, 0, 0, 0, 0, 0, 0, 0], "value": trace_id }
case _ =>
false
end
Rule FnCases
The FnCases rule defines a sequence of cases for structural pattern matching in tremor pattern functions.
rule FnCases ::=
FnCaseClauses FnCaseDefault
| FnCaseDefault
;
connector/console/in
The connector console via its standard in port
Rule FnCaseDefault
The FnCaseDefines rule defines a default match clause for use in pattern match function signatures in tremor.
rule FnCaseDefault ::=
'case' '_' Effectors
;
Rule FnCase
The FnCase rule defines an array predicate pattern supporting match clause for use in pattern match function signatures in tremor.
rule FnCase ::=
'case' '(' ArrayPredicatePatterns ')' WhenClause Effectors
;
A case definition in a pattern match function definition
use std::type;
fn snottify(s) of
# Matches the literal string "badger"
case ("badger") => "snot badger, hell yea!"
# Matches any well formed json argument
case (~ json||) => let s.snot = true; s
# Matches any literal string
case (s) when type::is_string(s) => "snot #{s}"
# Matches, everything else
case _ => "snot caller, you can't snottify that!"
end;
Rule FnCaseClauses
The FnCaseClauses defines the case syntax to structurally matched function signatures in tremor.
rule FnCaseClauses ::=
FnCase
| FnCaseClauses FnCase
;
Rule FnArgs
The FnArgs rule defines , comma delimited arguments to a tremor function.
rule FnArgs ::=
Ident
| FnArgs ',' Ident
;
Rule SimpleExprImut
The SimpleExprImut rule defines optionally parenthesized simple immutable expressions in tremor.
rule SimpleExprImut ::=
'(' ComplexExprImut ')'
| AlwaysImutExpr
;
A parenthetic or simple non-parenthetic expression
( 1 + 2 ) * 3;
Rule Literal
The Literal rule defines the set of primitive literals supported in tremor.
rule Literal ::=
Nil
| Bool
| Int
| Float
;
Primitive literal types in tremor are integer, floating point, boolean and the null reference.
String literals are UTF-8 encoded values and support interpolation with single line ( regular ) and multi line ( heredoc style ) variant forms.
Rule Nil
The Nil rule defines the syntax of the nil literal in tremor.
rule Nil ::=
'nil'
;
Example
null # The `null` literal value
Rule Bool
The Bool rule defines the syntax of boolean literal in tremor.
rule Bool ::=
'bool'
;
Example
true # The boolean `true` literal
false # The boolean `false` literal
Rule Int
The Int rule literal specifes the syntax of integer literals in tremor.
rule Int ::=
'int'
;
Integer literals
Integer literals in tremor are 64 bit signed values of the form
1234
Values can be separated by an _ ( underscore ) for easy reading
let million = 1_000_000
Rule Float
The Float rule literal specifes the syntax of IEEE float literals in tremor.
rule Float ::=
'float'
;
Floating point literals
Floating point literals in tremor are 64 bit signed IEEE floating point values of the form
1234.0e10
Values can be separated by an _ ( underscore ) for easy reading
let million = 1_000_000.1234e-5
Rule StringLiteral
The StringLiteral rule defines a string literal in tremor.
Strings are " single-quote or """ triple-quote delimited blocks of UTF-8 text.
A single-quote string is a single line string, supporting sting interpolation.
A triple-quote string is a multi-line string, supporting sting interpolation.
rule StringLiteral ::=
'heredoc_start' StrLitElements 'heredoc_end'
| '\\' StrLitElements '\\'
| '\\' '\\'
;
How do I define single line string literals?
A single line string MUST be on a single line with raw newline characters ( unless escaped ).
"I am a literal string"
How do i define multi line string literals?
A multi line string MUST span multiple lines with raw newline characters.
Multi line strings
"""
I am a
multi
line
string
"""
The following example is a malformed multi line string:
""" snot """
Which when executed will result in a compile time error:
Error:
1 | """ snot """
| ^^^ It looks like you have characters tailing the here doc opening, it needs to be followed by a newline
Simple and nested Interpolation
Strings in tremor can be interpolated with internal scripts
"""
I am an #{interpolated} #{event.sum / event.count} string
Interpolations can be simple, as above, or #{
merge event of
{ "#{snot}": """
#{badger * 1000 + crazy_snake }
""" }
}
"""
This will result in the output:
"\nI am an interpolated 5.0 string\n\nInterpolations can be simple, as above, or {\"sum\":10,\"count\":2,\"snot\":\"\\n 20001\\n\\n \"}\n"
Note that the merge operation merges an event { "sum": 10, "count": 10 } with in scope values of snot that evaluates to the literal string "snot" and the numerics badger ( 20 ) and crazy_snake ( 1 ). Interpolations are nestable and field names or any other string literal in tremor can be interpolated.
However, we do not recommend complex nested interpolated strings. Defining a function and calling it may be a better alternative for most applications and uses.
Rule StrLitElements
The StrLitElements rule defines the internal structure of a string literal in tremor.
String literal in tremor support string interpolation via the #{ and } escape
sequence. Content within the escape sequence can be any legal and valid tremor
expression.
rule StrLitElements ::=
StringPart StrLitElements
| '\\\\#' StrLitElements
| '#{' ExprImut '}' StrLitElements
| StringPart
| '\\\\#'
| '#{' ExprImut '}'
;
Inside a literal tremor string
A string literal in tremor is a composition of multiple segments or parts.
These can be composed of:
- One or many single line string parts
- One or many multi line string parts
- A blackslash escaped
\\#to escape interpolated syntax, optinally followed by more string literal parts - Or, a
#{..}delimited interpolated section- Within an interpolated section there are no constraints on raw newline usage
- For complex interpolated sections, prefer good indentation!
Rule StringPart
The StringPart rule defines a simple or heredoc style string part.
rule StringPart ::=
'string'
| 'heredoc'
;
The " delimited string is single line
The """ delimited string is multi line
The forms can be mixed to construct a string literal
Interpolation is supported by both forms
Rule List
The List rule defines a [ and ] square bracket delimited sequence of zero or many ',' delimited expressions.
rule List ::=
'[' ListElements ']'
| '[' ']'
;
How are literal array or list or vectorc data collections defined in tremor?
We use [ .. ] square brackets to delimit list-like data.
Are multi-dimensional arrays supported?
Multi-dimensional arrays are compositional and can be nested
Does tremor support typed lists?
No. A list in tremor can have elements from any supported primitive or structural type.
Validation that a list is for a single type - such as a list of boolean values can be defined as follows:
use std::type;
let bad = [true, false, "snot"];
let good = [true, false ];
fn list_of_bool(l) with
let valid = true;
for l of
case (i,e) when type::is_bool(e) and valid == true => let valid = true
case (i, otherwise) => let valid = false
end;
valid # return true if list is all bool, false otherwise
end;
list_of_bool(bad); # should fail
list_of_bool(good); # should succeed
This user defined function can then be used in guard clauses like when type::is_bool(e) ... in the
example code.
Rule ListElements
The ListElements rule defines a , comma delimited sequence of expression elements.
rule ListElements ::=
ListElements_
;
List literals
Unlike JSON, trailing commas are supported
[foo,] # A non empty list, a trailing comma is optionally permissible
Except in empty lists, where the idiomatic form is preferred:
[] # An empty list - no trailing comma here!
Rule ListElements_
The ListElements_ rule is internal to the ListElements rule.
The rule defines a sequence of , comma delimited expression elements using the Sep macro rule.
rule ListElements_ ::=
Sep!(ListElements_, ComplexExprImut, ",")
;
See ListElements rule for details.
This rule wraps away a lalrpop macro call for ease of reference in other rules in the grammar source.
Rule Record
The Record rule defines a set of name-value pairs delimited by , a comma.
Records are enclosed in { and } curly braces.
The record structure in tremor is backwards compatible with JSON.
All JSON records can be read by tremor.
Not all tremor records can be read by a JSON reader as tremor supports computations, comments and trailiing , commas
in its record and array structures.
rule Record ::=
'{' Fields '}'
| '{' '}'
;
How are literal records or map types or key-value collections defined in tremor?
We use { .. } squigly braces to delimit record -like data.
Can records be nested?
Record values can be any valid primitive of structural type supported by tremor, including other records.
Does tremor support typed records?
No. A record in tremor can have element values from any supported primitive or structural type in tremor.
Validation that a record conforms to a shape, schema or structure can be achieved through match expressions.
use std::type;
let bad = { "list-of-bool": [true, false, "snot"] };
let good = { "list-of-bool": [true, false ], "flag": true };
fn list_of_bool(l) with
let valid = true;
for l of
case (i,e) when type::is_bool(e) and valid == true => let valid = true
case (i, otherwise) => let valid = false
end;
valid # return true if list is all bool, false otherwise
end;
fn is_good_record(r) with
let valid = false;
match r of
case extract=%{ present flag, list-of-bool ~= %[] }
when list_of_bool(extract["list-of-bool"]) =>
let valid = true
case _ =>
let valid = false
end;
valid
end;
is_good_record(bad); # should fail
is_good_record(good); # should succeed
This user defined function can then be used in guard clauses like when type::is_bool(e) ... in the
example code.
Rule Field
The Field rule defines a : colon delimited name value pair for a record literal.
The name is a string literal.
The value is an expression.
rule Field ::=
StringLiteral ':' ComplexExprImut
;
How do i define record fields?
The syntax is similar to JSON:
{ "field": "value" }
With the exception that fields may have an optional ',' terminal trailing comma
Interpolated field names
As literal strings in tremor support string interpolation the following variants are equivalent:
{ "snot": "badger" }
let snot = "snot";
{ "#{snot}": "badger" };
But, not all legal variations are recommended:
let snot = """
snot"""; # This may result in tears
{ "#{snot}": "badger" };
Another legal but likely not useful variation:
let snot = { "snot": "badger" };
{ "#{snot}": "badger" };
Will result in a stringifield json being encoded as the field name:
{"{\"snot\":\"badger\"}":"badger"}
Rule Path
The Path rule defines path operations over expressions.
Path operations structures to be tersely indexed in a path like structure.
Path operations are supported on
- A subset of expressions ( record, array, function )
- Meta keywords like
$,args,state,event,group,window
rule Path ::=
MetaPath
| EventPath
| StatePath
| LocalPath
| ConstPath
| AggrPath
| ArgsPath
| ExprPath
;
Path expressions are how events and in memory state in tremor are referenced in scripts.
How to reference metadata
The tremor runtime can provide and consume metadata with the events being processed by the runtime.
Metadata is distinguished by a $ dollar sign
let foo = $request # The local path `foo` is a copy of the metadata `$request`
Connectors such as kafka and http can generate metadata that scripts, queries
and pipelines can manipulate and process to tune tremor's runtime behaviour.
How to reference the current streaming event
The current event streaming through tremor in the current pipeline will
be available to queries, logics and scripts via the event keyword.
The event keyword can be further dereferenced via path statements
# Where event is a record
let foo = event.snot; # The local path `foo` is a copy of the `snot` field from the current event
# Where event is an array
let foo = event[10]; The local path `foo` is a copy of the 10th element of the current event.
How to reference pipeline state
Scripts in tremor can store state that is available for the lifetime of a pipeline
via the state keyword.
The state keyword can be further dereferenced via path statements
# Where state is a record
let foo = state.snot; # The local path `foo` is a copy of the `snot` field from the state record
# Where state is an array
let foo = state[10]; The local path `foo` is a copy of the 10th element of the state array
### How to reference arguments
For operators and structures that support arguments the `args` keyword can be
used to dereference values via path statements.
```tremor
# Where state is a record
let foo = args.snot; # The local path `foo` is a copy of the `snot` field from the args record
Args are nominal and always record values in tremor.
How can window state be referenced
Operations supporting windows and groups can dereference the cached state via the
window and group keywords which both support path operations.
Rule ExprPathRoot
The ExprPathRoot rule defines a subset of expressions where path operations are supported.
These are:
- Record literals or references to records.
- Array literals or references to arrays.
- The result of function invocations.
- The result of Parenthetic expressions.
rule ExprPathRoot ::=
'(' ComplexExprImut ')'
| Invoke
| Record
| List
;
An internal rule to the ExprPath rule that defines legal roots for a dynamic path dereference.
Rule ExprPath
The ExprPath rule defines path operations for expressions.
rule ExprPath ::=
ExprPathRoot PathSegments
;
Allows dereferencing literal values vi path expressions
{"snot": 0, "badger": 1, "goose": 2}["badger"];
{"snot": 0, "badger": 1, "goose": 2}.badger;
...
some_record_fn().record_field
Rule MetaPath
The MetaPath rule defines path operations for event metadata references.
In the context of a streaming event, allows metadata generated by the runtime to be accessed via path operations.
It is also possible to write to metadata to hint at the runtime to perform certain functions on the event data being forwarded. Tremor operators and connectors can read and write metadata.
rule MetaPath ::=
'$' Ident PathSegments
| '$' Ident
| '$'
;
How do i reference event metadata?
Events in tremor encapsulate data from other systems sent and received from tremor via configured connectors. Information about that data or metadata can also be provided by the runtime, and used in some operators and connectors to control tremor's runtime behaviour.
Meta-data is accessed via the $ dollar symbol.
let metadata = $;
Metadata can be any legal tremor value, but it is typically a record structure
let metastring = "snot" + $;
Meta-data can be written through via a let operation
let $command = { "do-things": "with-this-meta-request" }
Rule AggrPath
The AggrPath rule defines path operations for group and window references.
In the context of a windowed operation, enables the group and window meta
keywords to partipcate in path operations.
rule AggrPath ::=
'group' PathSegments
| 'group'
| 'window' PathSegments
| 'window'
;
How do I reference the computed group dimension?
use std::record;
define window by_2 from tumbling
with
size = 2
end;
select {
"g": group[0], # Extract current group dimension
"c": aggr::stats::sum(event.c),
}
from in[by_2]
group by set(each(record::keys(event.g))) into out;
Rule ArgsPath
The ArgsPath rule defines path operations for args references.
rule ArgsPath ::=
'args' PathSegments
| 'args'
;
How do i reference arguments?
Arguments are encapsualted via the args keyword symbol.
let what = args;
Arguments are always record structured
1 | let args = 1;
| ^^^^^^^^ Can't assign to a constant expression
Arguments cannot be assigned to or overridden in scripts.
Rule LocalPath
The LocalPath rule enables path operations on locally scoped identifiers.
rule LocalPath ::=
Ident PathSegments
| Ident
;
A local path is simply a path structure that is bounded to a locally defined value
let snot = { "snot": "badger" };
snot.snot # The field 'snot', on the local path reference 'snot'
Rule ConstPath
The ConstPath rule enables path operations on module scoped references.
rule ConstPath ::=
ModPath '::' LocalPath
;
A fully scoped path that does not include dynamic resolution or runtime type information.
Rule StatePath
The StatePath rule defines path operations for user defined in memory state in tremor.
Allows the state value to be dereferenced via path operations.
rule StatePath ::=
'state' PathSegments
| 'state'
;
How do i reference state in tremor?
Tremor programs can be stateful in many ways - such as through the state managed by operators and windowed operations by the runtime on behalf of the user provided program.
The state keyword allows an arbitrary value controlled by a users program to be
maintained and managed by the user program.
let my_state = state;
State can be written through via a let operation
let state = match state of
case null => { "count": 1 }
case _ => { "count"": state.count + 1 }
end;
Rule EventPath
The EventPath rule defines path operations for streaming events in tremor.
Allows the current streaming event to be dereferenced via path operations.
rule EventPath ::=
'event' PathSegments
| 'event'
;
How do i dereference event data?
The current event is accessed via the event keyword.
The event can be any legal tremor value. If it is a record or an array, then it can be dereferenced via the path language in the usual way.
event.snot; # Event record, field 'snot'
let badger = event[0]; Event array, first element
Events can be mutated and manipulated and used as an output
select { "wrapped-event": event } from in into out;
Rule PathSegments
The PathSegments rule specifies the continuation of a path rule.
| Form Variation | Description |
|---|---|
.<Ident> | A terminal segment dereferencing a record field |
<Ident><PathSegments> | A non-terminal segment dereferencing a record field |
[<Selector>] | A range or index segment dereferencing an array |
[<Selector>] | A terminal range or index segment dereferencing an array |
[<Selector>]<PathSegments> | A non-terminal range or index segment dereferencing an array |
rule PathSegments ::=
'.' Ident PathSegments
| '[' Selector ']' PathSegments
| '[' Selector ']'
| '.' Ident
;
``tremor a.b a["b"] a[0..5]
Defines how structural values in tremor can be dereferenced to get at internal parts such as
an element of an array, or a field of a record.
## Rule Selector
The `Selector` rule specifies an index or range of an array.
A range is a `:` colon separated pair of expressions.
An index is a single expression.
```ebnf
rule Selector ::=
ComplexExprImut ':' ComplexExprImut
| ComplexExprImut
;
Field selection for records
Selecing a record field using array notation
let snot = badger["snot"]
Select the field 'snot' from the record 'badger'
Ordinal selection for arrays
let e = badger[0];
Select the 0th ( first ) element of the array 'badger'
Range selection for arrays
let e = badger[0:5];
Select the 0th up to but no including the 5th element of the array 'badger'
Rule Invoke
The Invoke rule specifies the syntax of a function invocation.
rule Invoke ::=
FunctionName '(' InvokeArgs ')'
| FunctionName '(' ')'
;
use std::string;
string::len("snot")
Rule FunctionName
The FunctionName rule defines a path to a function in tremor.
It can be an Ident for functions defined in local scope.
It can be a ModPath for functions in a modular scope.
rule FunctionName ::=
Ident
| ModPath '::' Ident
;
Rule ModPath
The ModPath rule defines a modular path.
A modular path is a sequence of Idents separated by a :: double-colon.
rule ModPath ::=
ModPath '::' Ident
| Ident
;
How do i reference something from the standard library?
The standard library contains reusable constants, functions
and other definitions that can be used in scripts via the
Use and ModPath rules.
For example, if you have a file called foo.tremor in a src
folder you can append this to your TREMOR_PATH environment
variable
export TREMOR_PATH=/path/to/src
Assuming foo.tremor contains the following code:
fn meaning_of_life() of
42
end;
We can use this in another script as follows:
use foo;
let meaning = foo::meaning_of_life();
Rule InvokeArgs
The InvokeArgs rule defines a sequence of expression statements.
rule InvokeArgs ::=
InvokeArgs_
;
A comma delimited sequence of complex expressions
Rule InvokeArgs_
The InvokeArgs_ rule is an internal rule of the InvokeArgs rule.
The rule specifies a ; semi-colon delimited sequence of expression statements.
rule InvokeArgs_ ::=
Sep!(InvokeArgs_, ComplexExprImut, ",")
;
See InvokeArgs rule for details.
This rule wraps away a lalrpop macro call for ease of reference in other rules in the grammar source.
Rule Drop
Drop halts event processing for the current event being processed returning control to the tremor runtime, dropping the event.
Constraints
The drop operation should be used with care as the in-flight event is
discarded by the runtime. Where circuit breakers, guaranteed delivery and
quality of service operations are being managed by the engine downstream
these should be carefully programmed so that drop operations have no
side-effects on non-functional behaviours of the tremor runtime.
Here be dragons!
rule Drop ::=
'drop'
;
define script boring
script
drop
end;
create script boring;
select event from in into boring;
select event from boring into out;
Drop signals to the tremor runtime that an event is not interesting and can be dropped without any further handling by the engine. Drop statements in a script or query result in the processing of the current event halting without any further action bye the tremor runtime.
The dropped event is discarded by the engine.
Rule Emit
Emit halts event processing for the current event being processed returning control to the tremor runtime, emitting a synthetic event as output.
By default, the emit operation will emit events to the standard output port out.
The operation can be redirected to an alternate output port.
rule Emit ::=
'emit' ComplexExprImut '=>' StringLiteral
| 'emit' ComplexExprImut
| 'emit' '=>' StringLiteral
| 'emit'
;
define script route
script
emit => "not_out"
end;
create script route;
select event from in into route;
select event from route/not_out into out;
Emit signals to the tremor runtime that an event has been processed fully and processing can stop at the point emit is invoked and a synthetic value returned without any further processing.
The emitted event is forwarded by the engine.
Rule Let
The Let rule allows an expression to be bound to a Path.
The Path references the subject of the assignment based on tremor's Path rules.
The bound Path is mutable.
rule Let ::=
'let' Assignment
;
How do i create a local mutable variable in tremor?
let my_var = "this is a string";
Rule Assignment
The Assignment rule allows an expression to be bound to a Path.
The Path references the subject of the assignment based on tremor's Path rules.
rule Assignment ::=
Path '=' SimpleExpr
;
The rule assigns an expression to a path by reference.
x = 2
Assigns x to the value literal 2
x.y = 2
Assigns the field y on record x to the value literal 2
Assignments expressions can be constant such as via the Const rule or mutable such as via the Let rule.
Rule Patch
The Patch rule defines the patch statement in tremor.
rule Patch ::=
'patch' ComplexExprImut 'of' PatchOperations 'end'
;
Patch insert a field in a record
let a = patch event of
insert "test" => 1
end;
Default patch templates
patch event of
default => {"snot": {"badger": "goose"}}
end
## Rule PatchOperations
The `PatchOperations` rule defines a sequence of semi-colon delimited patch operations.
```ebnf
rule PatchOperations ::=
PatchOperationClause
| PatchOperations ';' PatchOperationClause
;
A sequence of patch operations
Rule PatchField
The PatchField is a string literal identifying a the field of a record to which a PatchOperationClause is being applied.
rule PatchField ::=
StringLiteral
;
Rule PatchOperationClause
The PatchOperationClause rule defines operations of a patch statement.
A patch operation can:
- Insert, update, copy ( clone ), move ( rename ), merge or erase fields in a record.
- Apply a default operation on a field or on the whole input record.
rule PatchOperationClause ::=
'insert' PatchField '=>' ComplexExprImut
| 'upsert' PatchField '=>' ComplexExprImut
| 'update' PatchField '=>' ComplexExprImut
| 'erase' PatchField
| 'move' PatchField '=>' PatchField
| 'copy' PatchField '=>' PatchField
| 'merge' PatchField '=>' ComplexExprImut
| 'merge' '=>' ComplexExprImut
| 'default' PatchField '=>' ComplexExprImut
| 'default' '=>' ComplexExprImut
;
How do I insert a new value into a record?
patch {"a": 1, "b": 2, "c": 3 } of
insert "d" => "delta"
end;
It is a semantic error to insert a value if the field already exists:
Error in foo.tremor:2:3
1 | patch {"a": 1, "b": 2, "c": 3 } of
2 | insert "b" => "bravo"
| ^^^^^^^^^^^^^^^^^^^^^ The key that is supposed to be written to already exists: b
3 | end;
How do I update an existing value in a record?
patch {"a": 1, "b": 2, "c": 3 } of
update "b" => "bravo"
end;
It is a semantic error to update a value if the field does not already exist:
1 | patch {"a": 1, "b": 2, "c": 3 } of
2 | update "d" => "delta"
| ^^^^^^^^^^^^^^^^^^^^^ The key that is supposed to be updated does not exists: d
3 | end;
How do I insert or update a value in a record?
If the distinction between an insert and an update is not significant the
upsert operation will insert a new field, or update an existing field. This
operation is more flexible, but does not offer compile time errors to protect
against invalid usage. Where possible, use insert or update in preference
to upsert when a new field or replacing an existing fields value would be
an error given the business logic at hand.
How do I erase a field from a record?
patch {"a": 1, "b": 2, "c": 3 } of
erase "d"
end;
The field c is removed from our record
How do I rename a field?
patch {"a": 1, "b": 2, "c": 3 } of
move "c" => "d" # The value MUST be a string literal as it represents a field name
end;
The c field is removed and a d field is added with the value from c
How do I duplicate a field?
Similar to move, the copy operation copies the value of one field to a new field
patch {"a": 1, "b": 2, "c": 3 } of
copy "c" => "d" # The value MUST be a string literal as it represents a field name
end;
The c field is preserved, and the d field is added with a copy of the value from c
Can I use patch and merge together?
The merge operation in a patch expression can be applied to the patch
target record or to a specified field.
patch {"a": 1, "b": 2, "c": 3 } of
merge "d" => {}
end;
The field d is created in this case with the empty record.
patch {"a": 1, "b": 2, "c": 3 } of
merge => { "snot": "badger", "b": "bravo" } # This is a terse form of insert for `snot`, and `update for `b`
end;
Defaults
For repetitive or template operations, the default operation allows a patch record
to be defined that is effectively merged with the target document
patch event of
default => {"snot": {"badger": "goose"}}
end
If event is an empty record, the result is the same as the default value expression.
If event has a snot field, the value for snot is preserved. The default value is not used.
Like merge operations, the default merge operation can be very effective for reducing the
boilerplate complexity of patch operations and improving the readability of the transformations
being performed by readers of the code.
For example, we could limit our default record patch to apply to only the snot field as follows:
patch event of
default "snot" => {"badger": "goose"}
end
Rule Merge
The Merge rule defines a merge operation of two complex immutable expressions.
rule Merge ::=
'merge' ComplexExprImut 'of' ComplexExprImut 'end'
;
How do I merge two two records?
merge {"a": 1, "b": 2, "c": 3 } of
{ "b": "bravo", "c": "charlie", "d": "delta" }
end;
The merge expression loosely follows the semantics of RFC 7396 - JSON Merge Patch.
From our example:
- The field
ais not patched and preserved - The field
bis patched with the valuebravo- the original value is replaced - The field
dis not in the original, and is added.
The expression is useful when one record with another.
An alternative to the Merge expression is the Patch expression which is operation
rather than value based.
Rule For
The For rule defines an mutable for comprehension.
rule For ::=
'for' ComplexExprImut 'of' ForCaseClauses 'end'
| 'for' ComplexExprImut 'of' ForCaseClauses 'into' ComplexExprImut MaybeFoldOperator 'end'
;
An illustration of the rule is a set of case statements, followed by an optional default case
for event of
## Cases
end
Alternatively for expressions have an initial value and a fold operator. If neither is given the
initial value is [] and the operator + is used - so the above example can also be written as:
for event of
## Cases
into []
use +
end
For inserting a few keys into a record we could write
let base = {"snot": "badger"}
for new_values of
case (k,v) => {k: v}
into base
end
For creating the product of numbers in a array we use:
for event of
case (k,v) => v
into 1
use *
end
For a overview of supported operators see the operators section.
The ForCaseClause rule has examples of the two basic forms for record and array comprehensions.
Rule MaybeFoldOperator
rule MaybeFoldOperator ::=
( 'use' FoldOperator ) ?
;
Rule FoldOperator
rule FoldOperator ::=
BinOr
| BinXor
| BinAnd
| BinBitXor
| BinBitAnd
| BinBitShift
| BinAdd
| BinMul
;
Rule ForCaseClauses
The ForCaseClausest defines a sequence of case clauses in an mutable for comprehension.
rule ForCaseClauses ::=
ForCaseClause
| ForCaseClauses ForCaseClause
;
A set of one or many case clauses for record key-value or array index-element comprehensions
Rule ForCaseClause
The ForCaseClause defines the case clause for mutable for comprehensions.
rule ForCaseClause ::=
'case' '(' Ident ',' Ident ')' WhenClause Effectors
;
Record Comprehension
for { "snot": "badger" } of
case (name, value) => value
end;
Array Comprehension
for [1, "foo", 2, "bar"] of
case (index, value) => value
end;
Guards
use std::type;
for [1, "foo", 2, "bar"] of
case (index, value) when type::is_string(value) => { "string": value }
case (index, value) when type::is_integer(value) => { "integer": value }
end;
Rule ForImut
The ForImut rule defines an immutable for comprehension.
rule ForImut ::=
'for' ComplexExprImut 'of' ForCaseClausesImut 'end'
| 'for' ComplexExprImut 'of' ForCaseClausesImut 'into' ComplexExprImut MaybeFoldOperator 'end'
;
An illustration of the rule is a set of case statements, followed by an optional default case
for event of
## Cases
end
The ForCaseClauseImut rule has examples of the two basic forms for record and array comprehensions.
Rule ForCaseClausesImut
The ForCaseClausesImut defines a sequence of case clauses in an immutable for comprehension.
rule ForCaseClausesImut ::=
ForCaseClauseImut
| ForCaseClausesImut ForCaseClauseImut
;
A set of one or many case clauses for record key-value or array index-element comprehensions
Rule ForCaseClauseImut
The ForCaseClauseImut defines the case clause for immutable for comprehensions.
rule ForCaseClauseImut ::=
'case' '(' Ident ',' Ident ')' WhenClause EffectorsImut
;
Record Comprehension
for { "snot": "badger" } of
case (name, value) => value
end;
Array Comprehension
for [1, "foo", 2, "bar"] of
case (index, value) => value
end;
Guards
use std::type;
for [1, "foo", 2, "bar"] of
case (index, value) when type::is_string(value) => { "string": value }
case (index, value) when type::is_integer(value) => { "integer": value }
end;
Rule Match
The Match rule defines a mutable match statement in tremor.
rule Match ::=
'match' ComplexExprImut 'of' Predicates 'end'
;
An illustration of the rule is a set of case statements, followed by an optional default case
match event of
case %{} => "I am a possibly non-empty record",
case %[] => "I am a possibly non-empty array",
case %( "snot" ) => "I am a list with 1 element which is the string \"snot\"
case _ => "I'm something else"
end
Rule Predicates
The Predicates rule defines a sequence of mutable PredicateClause rules in tremor.
rule Predicates ::=
PredicateClause
| Predicates PredicateClause
;
An illustration of the rule is a set of case statements, followed by an optional default case
match event of
case %{} => "I am a possibly non-empty record",
case %[] => "I am a possibly non-empty array",
case %( "snot" ) => "I am a list with 1 element which is the string \"snot\"
case _ => "I'm something else"
end
Rule PredicateClause
The PredicateClause rule defines the forms of a mutable match statement in tremor.
rule PredicateClause ::=
'case' CasePattern WhenClause Effectors
;
How do i write robust match predicate rules?
There are two basic forms of match expression predicate
Case form
match event of
case %{} => "I am a possibly non-empty record",
case %[] => "I am a possibly non-empty array",
case %( "snot" ) => "I am a list with 1 element which is the string \"snot\"
These are used for isolating specific cases of interest that need to specific
processing. Sometimes cases can be incomplete without a default case and this
is also supported.
case _ => "If i'm not one of these things, I'm something else"
end
If tremor can prove at compile time that a default case is advisable it will
emit a warning:
match event of
case true => "I believe you"
end;
1 | match event of
2 | case true => "I believe you"
3 | end;
| ^^^ This match expression has no default clause, if the other clauses do not cover all possibilities this will lead to events being discarded with runtime errors.
And, tremor will issue a runtime error when a bad case is found:
1 | match event of
2 | case true => "I believe you"
3 | end;
| ^^^ A match expression executed but no clause matched
| NOTE: Consider adding a `case _ => null` clause at the end of your match or validate full coverage beforehand.
It is almost always preferable to have a default case and this practice is recommended.
Rule Effectors
The Effectors rule defines an effect block.
rule Effectors ::=
'=>' Block
;
Example
An effector is an => arrow followed by Block sequence.
In many of the structural forms such as match, for or patch and effector is
a sequence of logic that is executed when certain conditions occur. The final statement
in an effector is the result of the sequence of executions.
As an example, here is a for compresension that enumerates a list
and computes the stringified representation of the elements of the list.
The for expression collects each iterations result from the effector's block
statement and aggregates them into a list.
for [1, 2, 3, 4, 5] of
case (i, e) => "#{e}";
end;
Rule Block
The Block rule defines a semi-colon delimited set of Expr rules.
rule Block ::=
Expr
| Block ';' Expr
;
How does tremor process blocks?
A block of expressions is a ; semi-colon delimited set of statements that
share the same scope. This means that the same set of metadata, state and
any scoped identifiers are visible to the block.
The last expression in a block of statements is the return value.
let return = 1;
let return = return << 7 % 4;
return - 1
The answer as we're sure you'll agree is 7.
Rule MatchImut
The MatchImut rule defines a match statement in tremor.
rule MatchImut ::=
'match' ComplexExprImut 'of' PredicatesImut 'end'
;
An illustration of the rule is a set of case statements, followed by an optional default case
match event of
case %{} => "I am a possibly non-empty record",
case %[] => "I am a possibly non-empty array",
case %( "snot" ) => "I am a list with 1 element which is the string \"snot\"
case _ => "I'm something else"
end
Rule PredicatesImut
The PredicatesImut rule defines a sequence of PredicateClauseImut rules.
rule PredicatesImut ::=
PredicateClauseImut
| PredicatesImut PredicateClauseImut
;
An illustration of the rule is a set of case statements, followed by an optional default case
match event of
case %{} => "I am a possibly non-empty record",
case %[] => "I am a possibly non-empty array",
case %( "snot" ) => "I am a list with 1 element which is the string \"snot\"
case _ => "I'm something else"
end
Rule CasePattern
The CasePattern rule defines the valid structural pattern matching forms
available in a match statement's case clause.
rule CasePattern ::=
RecordPattern
| ArrayPattern
| TuplePattern
| ComplexExprImut
| '_'
| '~' TestExpr
| Ident '=' CasePattern
;
An internal part of the match statement rule supporting
- Record, Array and Tuple structural patterns
- Complex predicate expressions
- The "_" dont care condition
- A predicate extractor
- An aliased case pattern
Rule PredicateClauseImut
The PredicateClauseImut rule defines valid clauses of a match statement.
Two forms are supported:
- A
caseexpression with optional guard expression and mandatory effector block. - A
defaultcase expression with effector block.
rule PredicateClauseImut ::=
'case' CasePattern WhenClause EffectorsImut
;
How do i write robust match predicate rules?
There are two basic forms of match expression predicate
Case form
match event of
case %{} => "I am a possibly non-empty record",
case %[] => "I am a possibly non-empty array",
case %( "snot" ) => "I am a list with 1 element which is the string \"snot\"
These are used for isolating specific cases of interest that need to specific
processing. Sometimes cases can be incomplete without a default case and this
is also supported.
case _ => "If i'm not one of these things, I'm something else"
end
If tremor can prove at compile time that a default case is advisable it will
emit a warning:
match event of
case true => "I believe you"
end;
1 | match event of
2 | case true => "I believe you"
3 | end;
| ^^^ This match expression has no default clause, if the other clauses do not cover all possibilities this will lead to events being discarded with runtime errors.
And, tremor will issue a runtime error when a bad case is found:
1 | match event of
2 | case true => "I believe you"
3 | end;
| ^^^ A match expression executed but no clause matched
| NOTE: Consider adding a `case _ => null` clause at the end of your match or validate full coverage beforehand.
It is almost always preferable to have a default case and this practice is recommended.
Rule EffectorsImut
The EffectorsImut rule defines the result value block sequence of pattern rule.
The effectors block provides the result value of case and default clauses in
match statements, for comprehensions.
rule EffectorsImut ::=
'=>' BlockImut
;
Example
An effector is an => arrow followed by Block sequence.
In many of the structural forms such as match, for or patch and effector is
a sequence of logic that is executed when certain conditions occur. The final statement
in an effector is the result of the sequence of executions.
As an example, here is a for compresension that enumerates a list
and computes the stringified representation of the elements of the list.
The for expression collects each iterations result from the effector's block
statement and aggregates them into a list.
for [1, 2, 3, 4, 5] of
case (i, e) => "#{e}";
end;
Rule BlockImut
The BlockImut rule defines a comma delimited sequence of complex immutable expressions.
rule BlockImut ::=
ComplexExprImut
| BlockImut ',' ComplexExprImut
;
How does tremor process blocks?
A block of expressions is a ; semi-colon delimited set of statements that
share the same scope. This means that the same set of metadata, state and
any scoped identifiers are visible to the block.
The last expression in a block of statements is the return value.
let return = 1;
let return = return << 7 % 4;
return - 1
The answer as we're sure you'll agree is 7.
Rule WhenClause
The WhenClause rule defines an optional guard expression.
rule WhenClause ::=
( 'when' ComplexExprImut ) ?
;
Implicit guard
When no when clause is provided it is always executed equivalent to
when true
Explicit guards
Guards are predicate or boolean expressions, so any expression that reduces
to a boolean result can be used in the WhenClause
when present state.snot and present event.badger
In the above rule, the state must be a record with a field snot
present and the current event must be a record with a field badger present
Rule PredicateFieldPattern
The PredicateFieldPattern rule defines the legal predicate tests available
within record patterns.
Record patterns can use:
- Extractor test expressions against fields.
- Record, array and tuple patterns against fields.
- Equality and comparison predicate patterns against fields.
- Presence patterns against fields.
rule PredicateFieldPattern ::=
Ident '~=' TestExpr
| Ident '=' Ident '~=' TestExpr
| Ident '~=' RecordPattern
| Ident '~=' ArrayPattern
| Ident '~=' TuplePattern
| 'present' Ident
| 'absent' Ident
| Ident BinCmpEq ComplexExprImut
;
Extraction
x ~= json||
Tests if the field x represents an embedded json script. If true,
extracts and parses the embedded JSON and stores the parsed document
in the field x in the predicate patterns result, if configured.
Aliased extraction
alias = x ~= json||
Tests if the field x represents an embedded json script. If true,
extracts and parses the embedded JSON and stores the parsed document
in the field alias in the predicate patterns result, if configured.
Sub records
x ~= %{}
Tests if the field x represents a record value. If true, extracts embedded
record and stores it in the field x in the predicate patterns result, if configured.
Sub arrays
x ~= %[]
Tests if the field x represents an array value. If true, extracts embedded
array and stores it in the field x in the predicate patterns result, if configured.
x ~= %()
Tests if the field x represents an array value. If true, extracts embedded
array and stores it in the field x in the predicate patterns result, if configured.
Presence and absence
present x
Is the field x present in the record? If true, extracts the field.
absent x
Is the field x absent in the record?
Comprison and Equality tests
x >= 10
Tests if the numeric field x is greater than or equal to the literal 10. If true, extracts the field.
Rule TestExpr
The TestExpr defines an extractor with an optional microformat body.
A test expression has a predicate component. The Ident defines the
expected microformat the value being tested in a structural pattern
match should conform to.
If this validates, then an optional microformat expression that is
specific to the extractor named by the Ident is employed to extract
content from the value into a value that tremor can process.
rule TestExpr ::=
Ident TestLiteral
;
json||
The rule identifies an extractor by name and delimits any micro-format arguments using the '|' pipe symbol.
In the above example the extractor is a JSON recognizer that can detect well formed JSON embedded inside a string literal. Such values will match the test expression and be parsed so that the content can be used in logic expressions.
Rule RecordPattern
The RecordPattern defines structural patterns against record values.
Record patterns start with the %{ operator and end with '}'.
Patterns may be empty %{}, or a sequence of record pattern fields.
Record patterns are search oriented based on predicate matching.
Ordinal, order or position based matching in records is not defined.
rule RecordPattern ::=
'%{' PatternFields '}'
| '%{' '}'
;
An empty record pattern
Matches any record value
%{ }
A record with a field called 'snot'
%{ snot }
A record with a field called 'snot', with a string literal value 'badger'
%{ snot == "badger" }
A record with a field called 'snot', whose string contents is well-formed embedded JSON
%{ snot ~= json|| }
Rule ArrayPattern
The ArrayPattern defines structural patterns against array values.
Array patterns start with the %[ operator and end with ].
Patterns may be empty %[], or a sequence of array predicate patterns.
Array patterns are search oriented based on predicate matching.
Where ordinal matching is needed then a TuplePattern may be preferential.
rule ArrayPattern ::=
'%[' ArrayPredicatePatterns ']'
| '%[' ']'
;
match event of
case result = %[ 1, 2 ] => result
case %[ _ ] => "ignore"
case _ => null
end
Rule TuplePattern
The TuplePattern defines structural patterns against tuple values.
Tuple patterns start with the %( operator and end with ).
Patterns may be empty %(), %(...) any, or a sequence of tuple patterns
followed by an optional open tuple ... match.
Tuple patterns are ordinal patterns defined against arrays.
Where search like predicate filters are preferential the ArrayPattern may be a better choice.
rule TuplePattern ::=
'%(' TuplePredicatePatterns OpenTuple ')'
| '%(' ')'
| '%(' '.' '.' '.' ')'
;
A tuple pattern allows ordinal matching of arrays. A tuple pattern is useful
where the search oriented %[ .. ] array pattern syntax is insufficient or
when the order of entries or elements in an array is significant.
use std::string;
match string::split(event, "/") of
case %("snot") => 0 # An array with a single string literal 'snot' value
case %("snot", ...) => 1 # An array with a first value string literal 'snot', and possibly zero or many more values
case %("api", _, "badger", ...) => 2 # An array with first value 'api', and 3rd value 'badger'
case %("") => 3 # An array with an empty string literal value
case %("badger", "snot") => 4 The two element array with 1st element "badger", and 2nd element "snot"
case _ => string::split(event, "/")
end
Rule OpenTuple
The OpenTuple rule defines a tuple pattern that matches any element in a tuple
from the position it is used and subseuent elements.
It can only be used as an optional final predicate in a TuplePattern.
rule OpenTuple ::=
( ',' '.' '.' '.' ) ?
;
%("snot", ...)
An open tuple that starts with ["snot"] but may have zero or more extra elements
Rule TuplePredicatePatterns
The TuplePredicatePatterns rule defines a set of comma delimited TuplePredicatePattern rules.
rule TuplePredicatePatterns ::=
TuplePredicatePatterns ',' TuplePredicatePattern
| TuplePredicatePattern
;
See ArrayPredicatePattern rule for predicate patterns avilable in tuple predicate patterns.
The predicate patterns are essentially the same, with the distinction that tuple patterns are ordinal and array predicate patterns are search based.
Rule TuplePredicatePattern
The syntax of the TuplePredicatePattern is the same as that of the ArrayPredicatePattern.
rule TuplePredicatePattern ::=
ArrayPredicatePattern
;
Wraps an ArrayPredicatePattern as the predicates supported by Array and Tuple predicate patterns
are the same.
See the ArrayPredicatePattern rule for more details
Rule ArrayPredicatePattern
The ArrayPredicatePattern rule defines predicate patterns for structural pattern matching
against array values.
rule ArrayPredicatePattern ::=
'~' TestExpr
| '_'
| ComplexExprImut
| RecordPattern
;
Array predicate extraction - contains embedded JSON
~ json||
Contains records
%{}
Rule ArrayPredicatePatterns
The ArrayPredicatePatterns rule defines a set of comma delimited ArrayPredicatePattern rules.
rule ArrayPredicatePatterns ::=
ArrayPredicatePatterns ',' ArrayPredicatePattern
| ArrayPredicatePattern
;
Rule PatternFields
The PatternFields rule defines a set of comma delimited PredicateFieldPattern rules.
rule PatternFields ::=
PatternFields_
;
Wraps a macro invocation rule for for use in other rules.
Rule PatternFields_
The PatternFields_ rule is a rule that defines a comma separated set of PatternField definitions.
The rule follows the semantics defined in the Sep macro.
rule PatternFields_ ::=
Sep!(PatternFields_, PredicateFieldPattern, ",")
;
See PatternFields rule for details.
This rule wraps away a lalrpop macro call for ease of reference in other rules in the grammar source.
Rule Fields
The Fields rule defines a set of comma delimited Field rules.
rule Fields ::=
Fields_
;
Example for record patterns
%{ present snot, badger == "badger", list = %[] }
Rule Fields_
The Fields_ rule is a rule that defines a comma separated set of field definitions.
The rule follows the semantics defined in the Sep macro.
rule Fields_ ::=
Sep!(Fields_, Field, ",")
;
See Fields rule for details.
This rule wraps away a lalrpop macro call for ease of reference in other rules in the grammar source.
Rule Ident
An Ident is an identifier - a user defined name for a tremor value.
rule Ident ::=
'<ident>'
;
Examples of identifiers
let snot = { "snot": "badger" };
Keyword escaping
Surrounding an identifier with a tick '`' allows keywords in tremor's DSLs to be escaped
let `let` = 1234.5;
Emoji
You can even use emoji as identifiers via the escaping mechanism.
let `🚀` = "rocket";
But we cannot think of any good reason to do so!
Rule TestLiteral
The TestLiteral rule specifies an extractor microformat block.
An extractor takes the general form:
Ident '|' MicroFormat '|'
Where
The ident is the name of a builtin extractor such as json or base64.
The Microformat content depends on the extractor being used
rule TestLiteral ::=
'<extractor>'
;
Extracting JSON embedded within strings
let example = { "snot": "{\"snot\": \"badger\"" };
match example of
case extraction=%{ snot ~= json|| } => extraction.snot
case _ => "no match"
end;
When executed this will result in:
"badger"
Decoding base64 embedded within strings
let example = { "snot": "eyJzbm90IjogImJhZGdlciJ9Cg==" };
match example of
case extraction=%{ snot ~= base64|| } => extraction.snot
case _ => "no match"
end;
When executed this will result in:
"{\"snot\": \"badger\"}
Wrap and Extract
We can decode the base64 decoded string through composition:
let example = { "snot": "eyJzbm90IjogImJhZGdlciJ9Cg==" };
match example of
case decoded = %{ snot ~= base64|| } =>
match { "snot": decoded.snot } of
case json = %{ snot ~= json|| } => json.snot.snot
case _ => "no match - json"
end
case _ => "no match - base64"
end;
Rule BytesLiteral
The BytesLiteral is a representation of opaque binary data literals in tremor
The syntax is a subset of the bit syntax representation in the Erlang Programming Language.
We ❤️ Erlang.
We ❤️ bit syntax!
rule BytesLiteral ::=
'<<' '>>'
| '<<' Bytes '>>'
;
Examples
# Import standard tremor binary utility functions
use std::binary;
# Structure of a TCP packet header
# 0 1 2 3
# 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
# +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
# | Source Port | Destination Port |
# +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
# | Sequence Number |
# +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
# | Acknowledgment Number |
# +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
# | Offset| Res. | Flags | Window |
# +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
# | Checksum | Urgent Pointer |
# +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
# | Options | Padding |
# +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
# |
# Data section proceeding packet header is optional and may or |
# it is up to the user whether or not to include this in a |
# binary literal or not |
# |
# . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .|
# Record representation of a TCP packet
let event = {
"src": {"port": 1234},
"dst": {"port": 2345},
"seq": event,
"ack": 4567,
"offset": 1,
"res": 2,
"flags": 3,
"win": 4,
"checksum": 5,
"urgent": 6,
"data": "snot badger!" # For encoding we
};
# Convert the record into a binary encoded TCP packet
binary::into_bytes(<<
# Header segment
event.src.port:16, event.dst.port:16,
event.seq:32,
event.ack:32,
event.offset:4, event.res:4, event.flags:8, event.win:16,
event.checksum:16, event.urgent:16,
# Data segment
event.data/binary
>>)
Rule Bytes
The Bytes rule defines a sequence of bit syntax patterns in a binary tremor literal representation.
A legal sequence of bytes MUST contain at least one byte part segment.
Byte part segments are comma ( ',' ) delimited.
rule Bytes ::=
BytesPart
| Bytes ',' BytesPart
;
Example: How do I encode a TCP packet?
# Convert the record into a binary encoded TCP packet
binary::into_bytes(<<
# Encode source and destination TCP ports, each 16 bits wide
event.src.port:16, event.dst.port:16,
# Encode sequence, 32 bits wide
event.seq:32,
# Encode acknowldgement, 32 bits wide
event.ack:32,
# Encode TCP conditioning and flags fields
event.offset:4, event.res:4, event.flags:8, event.win:16,
# Encode checksum; and urgent bytes from first byte
event.checksum:16, event.urgent:16,
# Encode data using the encoded length of another binary literal
event.data/binary
>>)
Rule BytesPart
The BytesPart rule represents sub segment of a binary encoded literal
If the part is the last segment in a bytes literal, it can be of arbitrary length.
If the part is not the last segment, it must specify its length in bits.
rule BytesPart ::=
SimpleExprImut
| SimpleExprImut ':' 'int'
| SimpleExprImut '/' Ident
| SimpleExprImut ':' 'int' '/' Ident
;
Form
The part may take the following general form
SimpleExprImut ':' 'int' '/' Ident
Where:
- The `SimpleExprImut can be a literal or identifier to the data being encoded.
- A optional size in bits, or defaulted based on the data being encoded.
- An optional encoding hint as an identifier
Size constraints
The size must be zero or greater, up to and including but no larger than 64 bits.
Encoding Hints
| Ident | Description |
|---|---|
binary | Encoded in binary, using network ( big ) endian |
big-unsigned-integer | Unsigned integer encoding, big endian |
little-unsigned-integer | Unsigned integer encoding, little endian |
big-signed-integer | Signed integer encoding, big endian |
little-signed-integer | Signed integer encoding, little endian |
Rule Sep
The Sep rule is a LALRPOP convenience that allows defining
a macro rule template for a common
sub rule sequence.
The Sep macro rule definition in tremor DSLs allows lists or sequences of expressions to
be separated by a specified delimiter. The delimiter is optional for the final item in a list
or sequence.
| Argument | Description |
|---|---|
| T | The term rule - specifies what is to be separated |
| D | The delimiter rule - specifies how elements are separated |
| L | A list of accumulated terms |
macro Sep<L, T, D> ::=
T D L
| T D ?
;
Rule BinOp
The BinOp rule is a LALRPOP convenience that allows defining
a macro rule template for a common
sub rule sequence.
The BinOp macro rule definition in tremor DSLs allows binary operations to be defined tersely
| Argument | Description |
|---|---|
| Current | The current rule permissible for the LHS of the expression |
| Operation | The operation to be performeed |
| Next | The current rule permissible for the RHS of the expression |
The macro imposes rule precedence where the left hand side expression takes higher precedence relative to the right hand side expression when interpreted by tremor.
Considerations
Tremor performs compile time optimizations such as constant folding. So literal expressions
of the form 1 + 2 may compile to a constant ( 3 in this case ) and have no runtime cost.
macro BinOp<Op, Current, Next> ::=
( Current ) ( Op ) Next
;
Rule BooleanBinOp
macro BooleanBinOp<Op, Current, Next> ::=
( Current ) ( Op ) Next
;
Rule BinCmpEq
The BinCmpEq rule allows binary or comparative operations
Comparitive and Equality operations have the same precedence.
rule BinCmpEq ::=
BinEq
| BinCmp
;
Rule BinOr
The BinOr rule defines binary or operation
| Operator | Description |
|---|---|
xor | Binary or |
rule BinOr ::=
'or'
;
Logical OR
false or false # false
false or true # true
true or false # true
true or true # true
Rule BinXor
The BinXor rule defines binary exclusive or operation
| Operator | Description |
|---|---|
xor | Binary exlusive or |
rule BinXor ::=
'xor'
;
Logical Exclusive Or
false xor false # false
false xor true # true
true xor false # true
true xor true # false
Rule BinAnd
The BinAnd rule defines binary and operation
| Operator | Description |
|---|---|
and | Binary and |
rule BinAnd ::=
'and'
;
Logical And
false and false # false
false and true # false
true and false # false
true and true # true
Rule BinBitXor
The BinBitXor rule defines binary bitwise exlusive-or operation
| Operator | Description |
|---|---|
^ | Binary logical xor exclusive or |
rule BinBitXor ::=
'^'
;
Bitwise behaviour, when used with integers
42 ^ 2 # 40
42 ^ -2 # -44
42 ^ 0 # 42
-42 ^ 2 # -44
-42 ^ -2 # 40
Logical behaviour, when used with boolean predicates
true ^ true # false
true ^ false # true
Rule BinBitAnd
The BinBitAnd rule defines binary bitwise and operation
| Operator | Description |
|---|---|
& | Binary logical and |
rule BinBitAnd ::=
'&'
;
Bitwise behaviour, when used with integers
42 & 2 # 2
42 & -2 # 42
42 & 0 # 0
-42 & 2 # 2
-42 & -2 # -42
Logical behaviour, when used with boolean predicates
true & true # true
false & true, # false
Rule BinEq
The BinEq rule defines binary equality operations
| Operator | Description |
|---|---|
== | Binary equality |
!= | Binary non-equality |
rule BinEq ::=
'=='
| '!='
;
Are these values the same
let predicate = a == b
Are these values different
let predicate = a != b
Rule BinCmp
The BinCmp rule defines binary comparitive operations
| Operator | Description |
|---|---|
>= | Binary greater than or equal to |
> | Binary greater than |
<= | Binary less than or equal to |
< | Binary less than |
rule BinCmp ::=
'>='
| '>'
| '<='
| '<'
;
Is greater than
let predicate = a > b
Is greater than or or equal to
let predicate = a >= b
Is less than or equal to
let predicate = a <= b
Is less than
let predicate = a < b
Rule BinBitShift
The BinBitShift rule defines bit shift operations
| Operator | Description |
|---|---|
>>> | Binary bit shift right, unsigned |
>> | Binary bit shift right, signed |
<< | Binary bit shift left, signed |
rule BinBitShift ::=
'>>'
| '>>>'
| '<<'
;
[
"right bit shift (signed)",
42 >> 0, # 0
42 >> 2, # 10
-42 >> 2, # -11
42 >> 63, # 0
# right bit shift (signed) (invalid)
#42 >> 64,
#42 >> -1,
#42 >> 2.0,
#42 >> "2",
#42 >> true
"right bit shift (unsigned)",
42 >>> 0, # 42
42 >>> 2, # 10
-42 >>> 2, # 4611686018427387893
42 >>> 63, # 0
# right bit shift (unsigned) (invalid)
#42 >>> 64,
#42 >>> -1,
#42 >>> 2.0,
#42 >>> "2",
#42 >>> true
"left bit shift",
42 << 0, # 42
42 << 2, # 168
-42 << 2, # -168
42 << 63 # 0
# left bit shift (invalid)
#42 << 64
#a << 64
#42 << -1
#42 << 2.0
#42 << "2"
#a << "2"
#42 << true
#42 <<< 2
]
Rule BinAdd
The BinAdd rule defines additive operations
| Operator | Description |
|---|---|
+ | Binary addition |
- | Binary subtraction |
Note that the + binary operation is also used for string concatenation, array concatination and for iserting/overwriting fields in a LHS record from a HRS record.
rule BinAdd ::=
'+'
| '-'
;
Numeric addition
1 + 2 # Addition, if one operand is a float, the result will be a float
Numeric subtraction
1 - 2 # Subtraction, if one operand is a float, the result will be a float
Rule BinMul
The BinMul rule defines multiplicative operations
| Operator | Description |
|---|---|
* | Binary multiplication |
/ | Binary division |
% | Binary modulo |
rule BinMul ::=
'*'
| '/'
| '%'
;
Numeric multiplication
2 * 1.5 # multiplication, if any argument is floating point, result is floating point
Numeric division
2 / 1.5 # division, results is always a floating point value
Integer numeric Modulus
5 % 2 # modulus - operands must not integers