Unable to connect to splunk cloud from AKS cluster. time out error

[Yiyito11]

Hello All. Im trying to install the splunk conector inside our AKS cluster to collect app logs to send to our splunk cloud instance. I´m following this tutorial.

So I downloaded the chart, set the values.yaml with our splunk cloud instance and our token and run helm install command like this.

helm install my-collector .\splunk-otel-collector\ -n splunk with this values:

Daemonsets were deployed and pods are running, but if I check logs I´m seeing a timeout error .

So I got into the container a run a curl command to our splunk instance and was able to connect so no firewall or anything there in the middle.

What could I possible me missing here? is the url like in the example the correct way to configure this? I mean like:

https://mysplunkinstance.splunkcloud.com:8088/services/collector

The token is correct according to our config (otherwise I believe it would give a 401/403 or something in the error logs). In this case it´s trying to connect but without success with that timeout error

Anything else I need to be checking? I also change timeout from 10 to 20 in the values.yaml.

what header is expecting when it´s saying "exceeded while awaiting headers"?

There is no much about this issue on internet so I would appreciate any help.
Thank you!

[atoulme]

Please see this documentation to establish your HEC endpoint: https://docs.splunk.com/Documentation/Splunk/9.0.1/Data/UsetheHTTPEventCollector#Send_data_to_HTTP_Event_Collector_on_Splunk_Cloud_Platform

Please let us know if the issue still occurs - please consider opening a support case if you are stuck.

[atoulme]

AFAIK Splunk Cloud is not available on AKS. You need to consult with your Splunk admin to determine your HEC endpoint. It looks like you deployed Splunk with a BYOL (Bring Your Own License) set up. Please file a support case for more help if you are still unable to resolve the HEC endpoint.

[Yiyito11]

I´m not trying to deploy splunk in AKS, I´m trying to deploy Otel collector (which uses fluentd) to send logs to our cloud splunk instance.
and according to values.yaml in chart, it is available for AKS, EKS or GCP

[atoulme]

Understood. Please read the instructions in detail as they mention to use a separate host name for the HEC endpoint. If your Splunk Cloud instance is typically mysplunkinstance.splunkcloud.com, then your HEC endpoint hostname is http-inputs-mysplunkinstance.splunkcloud.com.

There are a couple ways to debug this further. You can curl the health API of the HEC endpoint with:
curl https://http-inputs-mysplunkinstance.splunkcloud.com/services/collector/health

You can also try to send a single HEC event with a curl command to check that everything is working with:
curl -k "https://http-inputs-mysplunkinstance.splunkcloud.com/services/collector/event" -H "Authorization: Splunk <TOKEN>" -d '{"event": "Hello, world!", "sourcetype": "manual"}'

Please make sure to try those out to check everything works as it should, and please open a support case if you need help.

[Yiyito11]

Thank you, will try and let you know

[Yiyito11]

OK it worked finally, thank you for your help @atoulme !