syslog source to loki sink - HTTP 400 Bad Request

[j4mbob]

Hi,

Im trying to ingest syslog messages from access points using the syslog source, filter for a specific message, then use loki as the source sink.

Tried various options but everytime loki just seems to respond with a HTTP 400 - Bad Request

heres my config:

[sources.aps]
type = "syslog"
address = "0.0.0.0:5143"
mode = "tcp"


[transforms.ap_connects]
type = "filter"
inputs = ["aps"]
condition = '.appname == "WIFI-4-CLIENT-CONNECTED"'

[sinks.loki]
type = "loki"
inputs = ["ap_connects"]
#remove_timestamp = true
endpoint = "http://loki:3100"

[sinks.loki.labels]
job = "vector"

[sinks.loki.encoding]
codec = "json"


[sinks.my_console]
type = "console"
inputs = [ "ap_connects" ]
target = "stdout"

[sinks.my_console.encoding]
codec = "json"

heres one of the syslog messages sent to stdout by the console sink, which is correctly showing only the transformed messages:

{"appname":"WIFI-4-CLIENT-CONNECTED","facility":"user","host":"zzz-zzz-zzz-zz-z","hostname":"blahblah","message":"Client [XX-XX-XX-XX-XX-XX] connected to WLAN [MY SSID] on radio [2]","severity":"warning","source_ip":"192.168.0.2","source_type":"syslog","timestamp":"2024-05-31T16:17:21Z"}

vector log output:

2024-05-31T15:18:32.122489Z DEBUG sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=35}:http: vector::internal_events::http_client: HTTP response. status=400 Bad Request version=HTTP/1.1 headers={"content-type": "text/plain; charset=utf-8", "x-content-type-options": "nosniff", "date": "Fri, 31 May 2024 15:18:32 GMT", "content-length": "201"} body=[201 bytes]
2024-05-31T15:18:32.122557Z ERROR sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=35}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being suppressed to avoid flooding.
2024-05-31T15:18:32.122591Z ERROR sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=35}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being suppressed to avoid flooding.
2024-05-31T15:18:32.122616Z ERROR sink{component_kind="sink" component_id=loki component_type=loki}:request{request_id=35}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.

I cant work out what i am doing wrong. using vector 0.38.0 and loki 2.8.0

any suggestions?

many thanks

Jamie

[tamer-hassan]

I had very similar issue (400 Bad requests from Loki) but with file source.
I installed Fiddler classic (proxy to intercept and monitor the request/response data) and configured the Loki sink with:

proxy.enabled = true
proxy.http = "http://localhost:8888" # Fiddler

and upon examining the response from Loki, it turned out it was complaining about timestamp too new due to different time zone setting. Applying the correct timezone to the time and reformatting it properly with parse_timestamp in Vector solved the issue.