Error: DNS server returned answer with no data

[elghazal-a]

Bug Report

Describe the bug
I use an http output to send logs to clickhouse but fluentbit can't resolve my domain clickhouse.internal.domain.com. What's special about this domain is that there a rewrite config in my coredns that rewrites the domain to an other domain that can be then resolved

here is the error log I get in fluentbit
[2024/06/21 16:56:16] [ warn] [net] getaddrinfo(host='clickhouse.internal.domain.com', err=1): DNS server returned answer with no data

To Reproduce

• install coredns
• configure a rewrite rule for specific domain
• install fluentbit in a container and instruct it to use coredns as dns server
• configure http output

This is example of my coredns config

.:53 {
  errors
  health {
    lameduck 5s
  }
  ready
  
  rewrite name regex clickhouse\.internal\.domain\.com             internal-proxy.internal-proxy.svc.cluster.local

  kubernetes cluster.local in-addr.arpa ip6.arpa {
    pods insecure
    fallthrough in-addr.arpa ip6.arpa
  }
  prometheus :9153
  forward . /etc/resolv.conf {
    max_concurrent 1000
  }
  cache 30
  loop
  reload
  loadbalance
}

This is an example of my fluentbit config

    [OUTPUT]
        name              http
        match             *
        host              clickhouse.internal.domain.com
        tls               off
        port              80
        http_user         fluentbit
        http_passwd       $$${FLUENTBIT_CH_PASSWORD}
        URI               /?query=INSERT+INTO+logs.traefik_internal+FORMAT+JSONEachRow
        format            json_stream
        json_date_key     date
        json_date_format  java_sql_timestamp

Your Environment
GKE
CoreDNS
fluentbit