Not able to use Fluentd custom plugin "[error]: config error file="fluent.conf" error_class=Fluent::NotFoundPluginError error="Unknown output plugin 'remove_tag'"

[Tamannaverma1912]

What is a problem?

We were trying to use fluentd cutsom plugin to make out plugin but fluentd running into NotFoundPluginError. And this behavior not just happened with the custom plugin but also with the build in plugin. Not really sure what we are missing and documentation itself is not very clear at all.
So for this we are using fluent/fluentd:edge-debian and for testing we ran our tests inside the fluentd docker container itself.

Describe the configuration of Fluentd

Firstly,
docker exec -it --user root fluentd-container /bin/bash

In this we have fluend directory at
`ls fluentd/plugins/

fluent-plugin-remove_tag fluent-plugin-test rewrite_tag_filter.rb`

But the documentation says that our plugin path should be at /etc/fluent/plugin but nothing related to fluentd is inside /etc/

Moving on Inside /fluentd/plugins/ for custom plugin we used
bundle gem fluent-plugin-remove_tag
& output was

      create  fluent-plugin-remove_tag/Gemfile
      create  fluent-plugin-remove_tag/lib/fluent/plugin/remove_tag.rb
      create  fluent-plugin-remove_tag/lib/fluent/plugin/remove_tag/version.rb
      create  fluent-plugin-remove_tag/sig/fluent/plugin/remove_tag.rbs
      create  fluent-plugin-remove_tag/fluent-plugin-remove_tag.gemspec
      create  fluent-plugin-remove_tag/Rakefile
      create  fluent-plugin-remove_tag/README.md
      create  fluent-plugin-remove_tag/bin/console
      create  fluent-plugin-remove_tag/bin/setup
      create  fluent-plugin-remove_tag/test/test_helper.rb
      create  fluent-plugin-remove_tag/test/fluent/plugin/remove_tag_test.rb
      create  fluent-plugin-remove_tag/.rubocop.yml

We modified fluent-plugin-remove_tag.gemspec to

require_relative "lib/fluent/plugin/remove_tag/version"

Gem::Specification.new do |spec|
  spec.name          = "fluent-plugin-remove-tag"
  spec.version       = "0.1.0"
  spec.authors       = ["Your Name"]
  spec.email         = ["[email protected]"]

  spec.summary       = %q{A custom Fluentd plugin to remove the tag field}
  spec.description   = %q{A simple custom Fluentd plugin for removing the tag field from log records}
  spec.homepage      = "https://example.com"

  spec.files         = Dir["lib/**/*.rb"]
  spec.require_paths = ["lib"]

  spec.add_dependency "fluentd", "~> 1.0"
end

and lib/fluent/plugin/output/remove_tag.rb to

require 'fluent/plugin/output'

module Fluent::Plugin
  class RemoveTagOutput < Fluent::Plugin::Output
    Fluent::Plugin.register_output('remove_tag', self)

    def configure(conf)
      super
    end

    def start
      super
    end

    def shutdown
      super
    end

    def process(tag, es)
      es.each do |time, record|
        record.delete('tag')
        router.emit(tag, time, record)
      end
    end
  end
end

and then ran gem build fluent-plugin-remove_tag.gemspec && gem install fluent-plugin-remove-tag-0.1.0.gem

Gem was successfully build and was there when we did
gem list

root@275799b63e65:/fluentd/plugins/fluent-plugin-remove_tag# gem list | grep remove
fluent-plugin-remove-tag (0.1.0)

and This is the fluent.conf we used

<source>
  @type tail
  path /test.log
  pos_file ./test.log.pos
  tag myapp.*
  <parse>
    @type json
  </parse>
</source>

<filter myapp.**>
  @type record_transformer
  enable_ruby true
  <record>
    namespace_name default
    pod_name test-pod
    container_name test-container
  </record>
</filter>

<filter myapp.**>
  @type record_transformer
  enable_ruby true
  <record>
    tag ${tag}
  </record>
</filter>

<match myapp.**>
  @type remove_tag
</match>

<match myapp.**>
  @type file
  path ./logs/namespace_name/pod_name/container_name/%Y-%m-%d-%H-%M.log
  <buffer namespace_name,pod_name,container_name,time>
    @type file
    path /var/log/fluentd-buffer
    flush_interval 10s
    chunk_limit_size 8m
    total_limit_size 512m
    timekey 1m
    timekey_wait 30s
    timekey_use_utc true
  </buffer>
</match>

This is basic config we just wanted to test the working of our custom plugin E2E and when we then ran

fluentd --config fluent.conf -p /fluentd/plugins/fluent-plugin-remove_tag/lib/fluent/plugin/

2024-05-28 14:10:35 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-05-28 14:10:35 +0000 [info]: parsing config file is succeeded path="fluent.conf"
2024-05-28 14:10:35 +0000 [info]: gem 'fluentd' version '1.17.0'
2024-05-28 14:10:35 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.5.2'
2024-05-28 14:10:35 +0000 [info]: gem 'fluent-plugin-remove-tag' version '0.1.0'
2024-05-28 14:10:35 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2024-05-28 14:10:35 +0000 [info]: gem 'fluentd' version '1.11.5'
2024-05-28 14:10:35 +0000 [error]: config error file="fluent.conf" error_class=Fluent::NotFoundPluginError error="Unknown output plugin 'remove_tag'. Run 'gem search -rd fluent-plugin' to find plugins"

Don't know what we missing here, Can anyone help. This error was not just with the custom plugin but built in plugin too
fluentd --config fluent.conf -p /fluentd/plugins/

2024-05-23 07:20:34 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-05-23 07:20:34 +0000 [info]: parsing config file is succeeded path="fluent.conf"
2024-05-23 07:20:34 +0000 [info]: gem 'fluentd' version '1.17.0'
2024-05-23 07:20:34 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.5.2'
2024-05-23 07:20:34 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2024-05-23 07:20:34 +0000 [info]: gem 'fluentd' version '1.11.5'
2024-05-23 07:20:34 +0000 [error]: config error file="fluent.conf" error_class=Fluent::NotFoundPluginError error="Unknown filter plugin 'rewrite_tag_filter'. Run 'gem search -rd fluent-plugin' to find plugins"

Describe the logs of Fluentd

No response

Environment

- Fluentd version: 1.17.0
- Fluent Package version:
- Docker image (tag): edge-debian
- Operating system: Linux 275799b63e65 5.15.49-linuxkit

[daipom]

@Tamannaverma1912 Thanks for detail report and sorry for the trouble.
I'll check this.

documentation itself is not very clear at all

Could you please tell me which documentation you saw?

[Tamannaverma1912]

https://docs.fluentd.org/plugin-development

[daipom]

Thanks!

[daipom]

I haven't been able to confirm this in detail yet, but it may have something to do with the following issue:

• -p overwrites the default plugin path #4269

In this case, you need to specify all plugin paths including the default path with multiple -p :
(Sorry, this is a known bug.)

fluentd --config fluent.conf  -p /fluentd/plugins/ -p /fluentd/plugins/fluent-plugin-remove_tag/lib/fluent/plugin/

[jashwantraj92]

@daipom Thanks for your response. Just want to make sure, the steps she has used to build the plugin is correct? If we dont want to use "-p" what would be the default path? The docs says /etc/fluent/plugin, but the docker image only has /fluentd folder.

[jashwantraj92]

@daipom WE figured out the problem. for plugins to be picked from plugin folder, the naming convention had to be correct.

"type_name". After changing the name to out_remove_metadata or formatter_remove_metadata, the custom plugins works.

[daipom]

Sorry for my late response.
I see! Thanks for your report.
Did the problem get resolved?

There is such a description in
https://docs.fluentd.org/plugin-development#installing-custom-plugins, but it may be inaccurate.
Maybe it should be improved.