Changelog

Unreleased

Breaking Changes

Introduce breaking change in kafka_consumer configuration, by adding mode.
Update grok to 0.2, optional patterns are now omitted no longer ""
Unify to url instead of endpoint for google and s3 connectors so they're in line with every other connector
Unify to urls for elastic connector
Unify to path for bench, kv and wal connector so it's in line with file
Rename s3_writer to s3_streamer

New features

Add gcl_writer Google Cloud Platform Cloud Logging connector for writing log entries
Add reverse functions and tests for arrays and strings, add sort test for arrays
Add a ClickHouse connector
Add a GCS streamer connector

Fixes

Fix invalid scope handling for flow definition default arguments
Fix elastic sink not acking events handled successfully if no source is connected.
Fix kafka_consumer possibly committing earlier offsets, thus replaying events that have already been handled.
Fix off-by-one error in kafka_consumer committing offsets, thus replaying the last committed event.
Allow kafka_consumer connector to reconnect upon more error conditions and avoid stalls.
Include flow alias in pipeline and connector aliases reported via metrics events and logging in order to deduplicate entries

[0.12.4]

Fixes

Fix startup startup script for systemd installations
Avoid hangs in elastic connector when not connected as source via its out or err port.
Default udp client bind ip to 0.0.0.0:0 instead of 127.0.0.1:0

New features

Add compression level for lz4, xz2 and zstd

[0.12.3]

Fixes

Don't fail unrecoverably when the connection to the server is lost in Google PubSub consumer.
Update bert operators to work with tremor 0.12

[0.12.2]

New features

Added the gpubsub_publisher connector
Added the gpubsub_consumer connector
Added new metadata options to elastic connector: version, version_type, retry_on_conflict, if_primary_term, if_seq_no

Fixes

Fix race condition leading to quiescence timeout when shutting down Tremor
Fix timeout config unit mismatch in qos::backpressure and qos::percentile operators. Changed to nanoseconds precision.
Update WAL operator to fix issue with overwriting first entries.

[0.12.1]

Fixes

Fix misbehaving stdio connector when stdio is a pipe
Fix Docker entrypoint script argument ordering
Fix id encoding for discord connector

0.12.0

New features

Add full TLS support for elastic connector (including client certificate auth)
Add several auth methods for elastic and HTTP connector (Basic Auth, Api-Key, Bearer Token, ...)
Add support for specifying client certificates with tls config for http_client and tcp_client connectors.
Add tremor new to create new template projects.
Support elastic.raw_payload for update
Add support for chunked requests and responses for the HTTP connectors
Add the gbq connector for Google BigQuery
Add support for tuple patterns inside of record patterns.
Refactor visitors to seperate walker and visitor and visit all nodes.
Add support for modular subqueries in Trickle
Add check_topic_metadata configuration flag to kafka source to bypass topic metadata fetch
port json! improvements to literal!
print names of failing tests
Add check_topic_metadata configuration flag to kafka source to bypass topic metadata fetch
port json! improvements to literal!
Add support for troy deployment language. The language adds the pipeline, connector, flow, links, connect, to, deploy and config reserved keywords. These are now reserved in the scripting and query languages and must now be escaped when they appear in event data. This is a breaking change in the query and script language dialects.
Add support for FLOAT4 and FLOAT8 serialization/deserialization to postgres connectors
Add std::path::try_default fn
Experimental ARM support
Add default => {...} and default "key" => "value" to patch
Add zstd pre- and post-processors #1100
Remove rental from Event #1031 #1037
Put event raw payload into Arc to improve cloning perf
Remove rental from the entire runtime #1037
Restructure operators and select to avoid transmutation #1024
Fix tremor-cli to use it's own binary when possible #1096
Restructure functions to avoid transmutation #1028
Update blackhole to print events / s #1129
Improve soundness and documentation of SRS code.
Add support for concatenating arrays #1113
Allow gcp headers to be included in rest offramp with auth: gcp
Add env source #1136
Remove need for emit_emtpy_window by automatic eviction for groups where no window holds data
Ensure all windows of a group are alligned and can not go out of sync
Box apropriately in the rest sink
replace macros with functions in gcp code
Add -q flag and clarify -v flag for unit tests
Add win::cardinality function
Backpressure now allows lossless (no discard) circuit breaker behaviour #1119
Allow functions, constants and expressions as roots for path lookups
Add the unix-socket source
Enabale automatic benchmakrs
Add the csv codec
Add the permissions option for setting the file mode for the unix-socket source
Tests can be run without their suite. #1238
Add the std::size module to convert sizes
Add custom function calls to constant folding
Integration test names are added as a tag so they can be run by name
integration tests low log stdout/stderr for before and after

Breaking Changes

the tremor api sub command in the cli has been removed.
the - is no longer a valid part of identifiers.
binaries now use _ to sepoerate type names as - is no longer a identifier.
changed naming for record object to avoid keywords like select and merge. New names are record.extract and record.combine.
command separators are now unified, both patch, match and for now use ; the same way the rest of the language does
in all definitional statements args now specifies interface arguments that are overwritable in the correspanding create statement, while with specifies non-overwritable configuration in both define and create statements - this unifies the use of with and args between trickle and troy
file connector no longer splits by lines - it now requires a preconnector
define for both troy and trickle now follow the same principle of define <type> <alias> from <source>
wal is no longer an operator but a connector
for the elastic connector indexes have not to be set on the batch not the individual event so one batch can only be to a single index.
metronome interval is now in nanoseconds (as all other timings)
Most connectors require a specified codec now instead of using JSON as a default
merge no longer treats null in the spec as a delete option but rather as a normal value
Combine all compression and decompression pre/postprocessors.

Fixes

Fix detection of *.troy files in entrypoint.sh causing duplicate configs to be loaded when using Kubernetes.
Fix a one-off error in the bench connector leading to it producing one event too much
Avoid acking events that failed while preprocessing or decoding using a codec.
Remove acknowledging events when dropped or sent to a dead end (e.g. unconnected port) as this was causing confusing and unwanted behaviour with error events
move EXIT connector to debug connectors to avoid shutting down tremor
Fix bug in grok extractor that would never return good matches
Properly terminate after tremor run ...
Fix the bench connector to actually stop after the given amount of events when iters is configured.
http headers to allow strings and arrays
fix the use of args in the with part of a create insode of a flow.
fix silent swallowing of unknown fields in connector definition.
Make otel severity_number optional: #1248
Don't allow duplicate stream names: #1212
Fix memory safety issue when using merge or patch with state as target and reassigning the resulting value to state #1217
Fix delayed event re-execution in case of errors in a branched pipeline
Skip instead of fail EQC on out of repo PRs
Ensure patch keys are strings to move runtime errors into the compiletime
Fix issue with the token " being presented as a tick in errors
Fix heredoc_start and heredoc_end showing up in error messages instead of """
Fix some errors in otel pb <-> json translation
Fix windowed queries emitting events with null metadata on tick
Fix sorting for artefacts
Fix issue where the test framework would generate reports without being asked for it #1072
Remove the need for eviction_period for time based windows
Remove dead code and unneeded allows in otel and gcp code
Fix test and suite names not being printed
Fix badly nested structure in unit tests
Fix bug in the unit testing framework that would ignore all tags
Fix the illogical structure of suites that required a doubly nested record
Fix -v flag
Fix issue with double counting of unit test stats
Fix issue with wrong script snippets being shown for unit tests
Fix argument order in test cases
Fix GCS go-auth token refresh
Fix create script syntax for aliased scripts with overridden params
Add benchmark names to benchmark tags
Kafka onramp: Remove failing metadata fetch in order to verify topic existance. Instead detect subscription errors and stop the onramp in that case.
Unix offramp: Add the missing StartStream message
tremor-script: Add more details about Unicode in the documentation of the string module
Fix hdr and dds aggregation function losing events when aggregating > 8192 events
Ensure merge can only happen on objects

0.11.12

Fixes

Unix onramp: Add the missing StartStream message to handle multiple connections correctly

0.11.10

Fixes

Fix windowed queries emitting events with null metadata on tick

0.11.9

Fixes

Kafka onramp: Remove failing metadata fetch in order to verify topic existance. Instead detect subscription errors and stop the onramp in that case.

0.11.8

Fixes

Fix hdr and dds aggregation function losing events when aggregating > 8192 events
Make otel severity_number optional: #1248

New Features

Add the unix-socket onramp

0.11.7

Fixes

Upgrade dependency on librdkafka to 1.6.1 #1228.

0.11.6

Fixes

Fix possible memory unsafety issue when using patch or merge on state #1217.

0.11.5

Fixes

Fix artefact sorting in tremor startup script

0.11.4

Update to clap 3, this forced some breaking changes:
- tremor server run -f file1 file2 now is tremor server run file1 file2
- tremor test -i i1 i2 -e e1 e2 is now tremor test -i i1 -i i2 -e e1 -e e2

New features

Add tests for patch feature in tremor-script #721
Add onramp for SSE (Server Sent Events) #885
Add raw flag in tremor-cli dbg #854

Fixes

Fix the Release archive and package build process
Replace preprocess from tremor-cli dbg with --preprocess option #1085

0.11.3

New features

Add tests for merge feature in tremor-script #721
Add support for receiving TLS encrypted data via TCP onramp.
Add support for sending TLS encrypted data via TCP offramp.

Fixes

Remove a number of transmutes
Catalog remaining transmutes with related tickets
Bump snmalloc-rs to 0.2.26 to ensure builds succeed on systems with libc < 2.25

0.11.2

New features

Add op key to KV offramp responses in order to differentiate responses by the command that triggered them
Change format of KV offramp responses to a more unified structure.
Add KnownKey::map_* functions to directly work on the Value::Objects inner HashMap, if available.
Add HEALTHCHECK to Dockerfiles
Improve printing for dot files
Scan sub directories of ${CFG_DIR} for config files.
Add offramp and onramp for AMQP with the lapin package.

Fixes

KV offramp sends error responses for each failed command
Ensure binding to first bind linked offramps to pipelines, then pipelines to offramps/onramps/pipeline, then onramps to pipelines to ensure events only start flowing when all downstreams are connected.
Fix empty record pattern to only match records.
Fix ws offramp not reconnecting after connection loss.
Run tests in tremor-cli bin
Switch operations of tremor dbg lex and tremor dbg preprocess as they did the job of the other.
Fix heredoc preprocessing, which was messing up error reporting
Fix false positives in cycle detection
Include cncf::otel stdlib sources in deb package
Add /usr/local/share/tremor to default TREMOR_PATH also for all packages as a well-known directory for custom tremor-script libraries and modules.
Record the partition number assigned during rebalancing when running Kafka.
Fix bug in HDR histogram implementation when using emit without reset.
Fix bug in mean that invalid values would be counted as part of the total number of values.
Avoid possible contraflow cycles via system::metrics pipeline, if a pipeline is connected to an output port of system::metrics pipeline.

0.11.1

New features

Add tremor_value::structurize convenience fn
Change qos::wal operator to only require one of max_elements or max_bytes (using both is still possible).
Add DNS sink
Add syslog codec.
Add $udp.host and $udp.port to allow controling udp packet destinations on a per event basis.
Deprecate udp.dst_* config, introduce udp.bind.* config instead.
Allow insights/contraflow events to traverse through multiple connected pipelines
Add GCP Cloud Storage linked sink connector.
Add textual-length-prefix pre and postprocessor.
Add GCP Pubsub sink and source connector.

Fixes

Inform docker builders that lack of resources may crash builds
Fix CI runners to work with caching
Fix dependencies for tremor-value and tremor-common
Fix docker entrypoint not forwarding arguments to tremor binary, unless -- is used before them
Fix match default clause only executing the last statement in the block.
Kafka back to async with a timeout on waiting
Fix qos::wal operator never cleaning up the last event from its storage
Fix generic::batch operator swallowing the transactional status of an Event
Fix windowed select queries not tracking the transactional status of an Event
Fix windowed select queries not tracking the Events that constitute an outgoing aggregated event
Avoid several offramps to swallow fail insights.
Send correlation metadata for send error events in elastic sink
Record the partition number assigned during rebalancing when running Kafka.

0.11.0

New features

CNCF OpenTelemetry source, sink and cncf::otel tremor-script library adds log, trace, metrics OpenTelemetry support
Fixes module path function resolution for tremor-cli doc tool to use unified path resolution
Removed the vsn.sh script which checks if the lockfile is up to date and replaces it with the --locked flag #798
Allow using '_' as seperators in numeric literals #645
Refactor kafka metadata variables to be under a single record $kafka.
Add support for Kafka message headers, available through the $kafka.headers metadata variable.
Add the cb offramp for testing upstream circuit breaker behaviour #779
Add the kafka onramp config retry_failed_events to acoid retrying failed events, and polling_interval to control how often kafka is polled for new messages if none were available previously #779
Add kv connector with the supported operations put, get, delete, scan, cas.
Add discord badge to README.md.
Handle signals and terminate properly on Ctrl+C in docker #806
Update to rust 1.50.0
Emit error events to the err port on exceeding concurrent requests limit for rest and elastic offramps.
Add the max_groups and emit_empty_windows settings on window definitions #828
Restrict event and event metadata references in the SELECT clause of a windowed select statement #828
Improve default visibility of tremor info logs from packages #850
Include the request info of a response for a linked rest offramp, available through the $response.request metadata variable.
Emit warnings when a window with emit_empty_windows without guards is used.
Extend match to handle top-level ~ extractors in match #834
Extend match to allow more assign expressions (i.e. case a = ~ glob|snot*| => or case v = _ =>)
Match pipeline improvements: tree search for ==, grouping based on shared keys, re-ordering of exclusive case statements
Optimize glob matches of the form glob|snot*| or glob|*badger| to cheaper prefix and suffix checks
Remove warnings for match w/o default if a _ case or a v = _ case exists
Add --exprs-only to dbg ast to not show metadata
Add Delete and Update to ES sink #822
Add more metadata to Kafka source #874
Update to simd-json 0.4
Add tests covering basic operations and string interpolation for tremor-script#721
Add offramp and onramp for NATS.io.
Add a stdin onramp.
Add tests covering arrays and records for tremor-script#721
Support for non caching UDP sink #900

Fixes

Fix kafka onramp hanging with no message in the queue, leading to delayed offset commits #779
Fail the kafka onramp if any of the configured topics could not be subscribed to #779
Tremor no longer requires a home dir for operations that do not need a config #782
Add performance section to PR template.
Fix markdown for discord link.
Fix lalrpop builds when extra folders exist.
Refactor operator metrics collection to eliminate clones in most cases.
Fix systemd spec file to load tremor files in /etc/tremor/config #784
Do not crash when we can not execute a config file #792
Sort the artefacts while running the benchmarks so that the benchmark run is more deterministic #825
Remove the bench_pipe_passthrough_csv benchmark #825
Fix a bug in the test bench command that was giving false negatives when the benchmarks were failing #816
Fix time based windows to not emit empty windows anymore and to not error if they contain event references #828
Do not commit an empty topic-partition-list in the kafka onramp and improve logging for better debugging
Fix kafka consumer offset lag of at least 1 continually by using offset + 1 when committing.
Fix issue where binary not (!) was not getting lexed correctly #833
Fix missing ack/fail insight events with offramps that dont support guaranteed delivery (e.g. udp, stdout) #870
Fix wrong error message for misconfigured UDP sinks
the api-host argument is renamed from -h to -a to resolve a conflict with -h being used for help

0.10.2

Fixes

Ensure blaster sends all events from the source #759
Allow the use of const and custom functions using const in select queries #749
Print hygenic errors when invalid trickle files are loaded in server run -f ... #761
Ensure elastic offramp does not issue empty bulk requests.
Avoid sending empty batches from the batch operator.
Ensure elastic offramp includes event payload in every error response.

New features

Add discord connector to allow communicating with the discord API

0.10.1

Fixes

Update tremor-value to 0.2 to include binary changes and thus unbreak the 0.10 tremor-script crate

0.10.0

Breaking Changes

String interpolation is now done via #{..} instead of {}. { no longer needs to be escaped, but \#{ needs an escape for literal #{.
Emit tumbling windows based on size immediately when they are full. #731
Emit tumbling windows based on time interval also when no event comes in but the interval passed #731
Elasticsearch offramp elastic: change the config value endpoints to nodes #732

New features

Default to thin-lto for all builds (prior this was only done in docker)
Automatically generate rpms and tarballs for releases.
Update rust to 1.49.0
Build deb packages
Statically link openssl
elastic sink now supports linked transports #715

Fixes

rewrite string interpolation to fix #726

0.9.4

New features

Extract simd_json::BorrowedValue into tremor specific tremor-value to allow extension of the type system.
Introduce Binary datatype and binary semi literals << 1, 2, event, (3 + 7) >>
Introduce base64 module for encoding and decoding binary to base64 encoded strings
introduce binary module to work with binary data

Fixes

Terminate pipeline creation when a node already exists with the given name #650
Fix visibility of pipeline metrics #648
Fix panic upon usage of postgres ramps due to incompatbility with tokio and async-std runtime. #641
Fix missing newlines when error line doesnt end with a newline #676
Fix possible panic in random::integer and random::float when used with values <= 0. #679

0.9.3

New features

Add prefix and raw config options to stdout and stderr offramps #637

Fixes

Make use of postprocessors in stdout, stderr and udp sinks #637
Allow to express minimal value of i64 as int literal #629
Fix scientific float literals (e.g. 1.0e-5) #629
Output errors to stderr on tremor run invocation. #629
Output more helpful errors for runtime errors (like accessing non-existing fields) #629
Fix gelf preprocessor to accept valid (unchunked) gelf messages #628
Fix memory access issue for large objects

0.9.2

New Features

Add docker image, and action for training docker image #576
Performance enhancements #608

Breaking Changes

Standardize error responses for tremor api #559
Remove empty event filtering from lines preprocessor logic #598

Fixes

Fix possible crashes from todo macro in tcp sink #573
Fix linked offramp not shutting down after binding is deleted #582
Fix slow kafka sink when queue.buffering.max.ms is set to > 0 #585
Fix string and heredoc errors with and without interpolation #595
Add hygienic error feedback for tremor run #620
Fix Kafka SmolRuntime hang #558
Remove array access to prevent possible runtime panics #574, #598
Update CLI: -i allows selecting a subset now #580
Allow using err for errors in tremor run #592
Update to rust toolchain 1.48

Files

CHANGELOG.md

Latest commit

History

CHANGELOG.md

File metadata and controls

Changelog

Unreleased

Breaking Changes

New features

Fixes

[0.12.4]

Fixes

New features

[0.12.3]

Fixes

[0.12.2]

New features

Fixes

[0.12.1]

Fixes

New features

Breaking Changes

Fixes

Fixes

Fixes

Fixes

Fixes

New Features

Fixes

Fixes

Fixes

New features

Fixes

New features

Fixes

New features

Fixes

New features

Fixes

New features

Fixes

Fixes

New features

Fixes

Breaking Changes

New features

Fixes

New features

Fixes

New features

Fixes

New Features

Breaking Changes

Fixes