Can't access tag from record accessor in rewrite_tag rule

[braydonk]

Bug Report

Describe the bug
Due to the way the rewrite_tag plugin does a record accessor check, using $TAG as the KEY in the rewrite_tag rule does not work. As far as I can tell, it is not possible to refer to the tag in the KEY part of the rule.

To Reproduce

[INPUT]
    name dummy
    # dummy {"tag": "x"}
    tag x

[FILTER]
    name rewrite_tag
    # rule $tag ^(x)$ y true
    rule $TAG ^(x)$ y true
    match x

[OUTPUT]
    match *
    name stdout

Expected behavior
This should produce 2 logs each dummy interval, one with tag x and one with tag y. With either rule in rewrite_tag, it does not work. The result is the same:

$ fluent-bit -c rewrite_tag.conf 
Fluent Bit v3.0.4
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/06/22 13:52:21] [ info] [fluent bit] version=3.0.4, commit=ce7aafab40, pid=11718
[2024/06/22 13:52:21] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/06/22 13:52:21] [ info] [cmetrics] version=0.9.0
[2024/06/22 13:52:21] [ info] [ctraces ] version=0.5.1
[2024/06/22 13:52:21] [ info] [input:dummy:dummy.0] initializing
[2024/06/22 13:52:21] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/06/22 13:52:21] [ info] [input:emitter:emitter_for_rewrite_tag.0] initializing
[2024/06/22 13:52:21] [ info] [input:emitter:emitter_for_rewrite_tag.0] storage_strategy='memory' (memory only)
[2024/06/22 13:52:21] [ info] [sp] stream processor started
[2024/06/22 13:52:21] [ info] [output:stdout:stdout.0] worker #0 started
[0] x: [[1719078742.640661254, {}], {"message"=>"dummy"}]
[0] x: [[1719078743.640598119, {}], {"message"=>"dummy"}]
[0] x: [[1719078744.640609670, {}], {"message"=>"dummy"}]
[0] x: [[1719078745.640618856, {}], {"message"=>"dummy"}]
[0] x: [[1719078746.640703880, {}], {"message"=>"dummy"}]
[0] x: [[1719078747.640635564, {}], {"message"=>"dummy"}]
[0] x: [[1719078748.640610046, {}], {"message"=>"dummy"}]

I have both $TAG and $tag as options because I have seen people reference $tag as the way to refer to the tag of the log record itself, though skimming through the docs I am having trouble finding that mentioned anywhere. In either case though, both don't work.

As a test, I also tried a dummy log with tag as a key, and used the $tag rule. This works as expected:

$ fluent-bit -c rewrite_tag.conf 
Fluent Bit v3.0.4
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/06/22 13:55:02] [ info] [fluent bit] version=3.0.4, commit=ce7aafab40, pid=11736
[2024/06/22 13:55:02] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/06/22 13:55:02] [ info] [cmetrics] version=0.9.0
[2024/06/22 13:55:02] [ info] [ctraces ] version=0.5.1
[2024/06/22 13:55:02] [ info] [input:dummy:dummy.0] initializing
[2024/06/22 13:55:02] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/06/22 13:55:02] [ info] [input:emitter:emitter_for_rewrite_tag.0] initializing
[2024/06/22 13:55:02] [ info] [input:emitter:emitter_for_rewrite_tag.0] storage_strategy='memory' (memory only)
[2024/06/22 13:55:02] [ info] [sp] stream processor started
[2024/06/22 13:55:02] [ info] [output:stdout:stdout.0] worker #0 started
[0] x: [[1719078903.640653219, {}], {"tag"=>"x"}]
[0] y: [[1719078903.640653219, {}], {"tag"=>"x"}]
[0] x: [[1719078904.640613005, {}], {"tag"=>"x"}]
[0] y: [[1719078904.640613005, {}], {"tag"=>"x"}]
[0] x: [[1719078905.640582523, {}], {"tag"=>"x"}]
[0] y: [[1719078905.640582523, {}], {"tag"=>"x"}]
[0] x: [[1719078906.640629124, {}], {"tag"=>"x"}]
[0] y: [[1719078906.640629124, {}], {"tag"=>"x"}]
[0] x: [[1719078907.640659315, {}], {"tag"=>"x"}]
[0] y: [[1719078907.640659315, {}], {"tag"=>"x"}]

Screenshots

Your Environment

• Version used: 3.0.4
• Configuration: Pasted in repro steps
• Environment name and version (e.g. Kubernetes? What version?): Ubuntu 22.04
• Server type and version: Ubuntu 22.04
• Operating System and version: Ubuntu 22.04
• Filters and plugins: rewrite_tag

Additional context
This seems to happen because the flb_regex_ra_match is only run on the log body:

fluent-bit/plugins/filter_rewrite_tag/rewrite_tag.c

Lines 356 to 381 in f5737c0

	static int process_record(const char *tag, int tag_len, msgpack_object map,
	const void *buf, size_t buf_size, int *keep,
	struct flb_rewrite_tag *ctx, int *matched,
	struct flb_input_instance *i_ins)
	{
	int ret;
	flb_sds_t out_tag;
	struct mk_list *head;
	struct rewrite_rule *rule = NULL;
	struct flb_regex_search result = {0};
	if (matched == NULL) {
	return FLB_FALSE;
	}
	*matched = FLB_FALSE;
	mk_list_foreach(head, &ctx->rules) {
	rule = mk_list_entry(head, struct rewrite_rule, _head);
	if (rule) {
	*keep = rule->keep_record;
	}
	ret = flb_ra_regex_match(rule->ra_key, map, rule->regex, &result);
	if (ret < 0) { /* no match */
	rule = NULL;
	continue;
	}

Based on other usages of flb_ra_regex_match, perhaps this is just intended behaviour and this isn't meant to be possible. If that is the case, then this the outcome for this issue is probably just docs, because based on previous interactions I don't think I'm the only user confused by this.