Need help configuring Vector to transform nested JSON

[dimidevops]

Hi all,

I'm running Vector(0.38.0) + Loki(3.0.0) on vanilla k8s to experiment with the tech. I have some demo apps deployed that are generating logs but I cannot seem to find a way to transform(parse) the JSON from the app logs and extract the all the json objects as labels so it would be easier to search in Grafana. I've tried all the ways that are mentioned in the documentation about json transformations but none of them seem to work with my case.

I've attached my config.yaml and few raw json logs samples

config.txt

raw_log_http_request_sample.json
raw_log_http_response_sample.json
raw_log_http_response_sample1.json

Any help or a nudge in the right direction is much appreciated

Thank you.

[jszwedko]

If you haven't already seen it, https://playground.vrl.dev/, can be useful to test out VRL programs for transformation. parse_json can be used to parse the input as JSON.

What do you mean by:

extract the all the json objects as labels so it would be easier to search in Grafana

?

[dimidevops]

Thank you, I haven't seen the playground and it is really helpful but now I can show you the issue I'm facing.

I'm trying to parse the response json from one of my samples using this VRL
As you can see it parses the original message but not the nested message inside

Attached is a pretty formatted fully parsed json from one of the response samples and I want to be able to extract certian values for example:

`.kubernetes.pod_name

.message.app_message.correlation
`
The goal is to be able to parse any json values from the original ingested event and send it to Loki sink as a transformed json with only selected needed values instead of the entire raw log.

Hope this makes sense

[jszwedko]

I think you may want to do . = merge!(parse_json(del(.message)) to parse the inner message field and merge the result into the top-level event.

[dimidevops]

Doesn't seem to work on the playground with any of my samples.

I guess the best solution would be to rewrite the logging output of the apps and configure separate source/sink for them

[jszwedko]

The error message there includes pointers on how to resolve it. In this case, I think the simplest way is using parse_json! (with a !) to indicate that the VRL program should abort if the input isn't valid JSON.