ECS metadata fails when container tag is overriden

[RaJiska]

Bug Report

Describe the bug
ECS metadata plugins fails to retrieve task details when the tag is overridden (by default the container ID) for something custom.

To Reproduce
Config file:

[SERVICE]
  daemon    Off
  log_level info
  flush     30

[INPUT]
  Name   forward
  Listen 0.0.0.0
  Port   ${AGENT_PORT}

[FILTER]
  Name                           ecs
  Match                          ecs.*
  ADD ecs_cluster                $ClusterName
  ADD ecs_task_id                $TaskID
  ADD ecs_container_name         $ECSContainerName
  ADD ecs_task_definition_family $TaskDefinitionFamily
  ADD ecs_task_id                $TaskID

[OUTPUT]
  Name  stdout
  Match ecs.*

Container sending logs started with the following command:

docker run --rm -it --log-driver=fluentd --log-opt fluentd-address=localhost:24224 --log-opt "tag=ecs.test123456" alpine echo lol

Resulting in the following errors from fluentbit:

[2024/07/01 09:38:49] [ warn] [filter:ecs:ecs.1] Failed to get metadata from /v1/tasks?dockerid=ecs.test1234, will retry
[2024/07/01 09:38:49] [ info] [filter:ecs:ecs.1] Requesting metadata from ECS Agent introspection endpoint failed for tag ecs.test123456

Expected behavior
Fluent-bit ECS filter should not retrieve the container ID from the tag. I am not aware of the internals, but I believe there must be another way to retrieve the container ID as it is available through the container_id key regardless.

Your Environment

• Version used: fluent/fluent-bit:3.0.7
• Configuration: Described above
• Environment name and version (e.g. Kubernetes? What version?): docker
• Filters and plugins: ecs

Additional context
I'd like to supply a custom tag to have a different logic depending of the container and log types it ships to fluentbit. The ECS metadata filter does not work when supplying my own tag.