Redis queries are not registered in spans (db.statement)

[dasivon021]

We're using Spring Boot 3.2.3 with spring-lettucemod to issue RediSearch (full-text) queries. We also use javaagent for exporting logs, metrics & traces to the OpenTelemetry backend. For dev/local purposes we're using grafana/otel-lgtm image. Reproducible example with step-by-step instructions is here.

The problem occurs when I want to see spans and want to isolate exact Redis query (db statement) that is issued. All I can see in Tempo is db.statement "SEARCH ? ? ? ? ? ? ? ?". Actual Redis query looks something like this: "FT.SEARCH" "myIdx" "@User\\-Name:{John}" "LIMIT" "0" "10000" "SORTBY" "User-Name" "ASC"

I can see that spring-lettucemod transitively includes lettuce:lettuce-core redis driver:

[INFO] +- com.redis:spring-lettucemod:jar:3.8.0:compile
[INFO] |  +- com.redis:lettucemod:jar:3.8.0:compile
[INFO] |  |  \- **io.lettuce:lettuce-core:jar:6.3.1.RELEASE:compile**
...
[INFO] |  \- org.apache.commons:commons-pool2:jar:2.12.0:compile 

According to this docs, Lettuce is supported for database client spans. So, if I'm correct, these Redis queries should be visible in Tempo.

I've also found these two system properties for java agent.

So my full command for starting the Spring Boot app looks like this:

java -javaagent:.\target\lib\opentelemetry-javaagent.jar -jar .\target\demo-0.0.1-SNAPSHOT.jar -Dotel.traces.exporter=otlp -Dotel.metrics.exporter=otlp -Dotel.logs.exporter=otlp -Dotel.instrumentation.lettuce.experimental-span-attributes=true -Dotel.instrumentation.lettuce.connection-telemetry.enabled=true -Dotel.instrumentation.common.default-enabled=true -Dotel.instrumentation.lettuce.enabled=true -Dotel.service.name=demo-otel-redis -Dotel.exporter.endpoint=http://localhost:4317

When I open Grafana, this is what I see:

On this list, I can see that some attributes are correctly set, such as db.system="redis". But some are missing such as db.query.text and db.query.parameter.<key>.

I'm not sure why db.statement is even appearing as a span attribute when it is deprecated. Is it because of the image/backend we're using or is there some other property that needs to turned on/off ? Just as a side note, If I use Hibernate with Postgres db, query is visible in the spans.

So, in short, how to enable Redis queries (with parameter values) in Tempo using javaagent and otel?

[laurit]

The problem occurs when I want to see spans and want to isolate exact Redis query (db statement) that is issued. All I can see in Tempo is db.statement "SEARCH ? ? ? ? ? ? ? ?". Actual Redis query looks something like this: "FT.SEARCH" "myIdx" "@User\\-Name:{John}" "LIMIT" "0" "10000" "SORTBY" "User-Name" "ASC"

By default redis queries are sanitized, this can be disabled with -Dotel.instrumentation.common.db-statement-sanitizer.enabled=false. This is done so because queries could potentially contains sensitive components like PII, medical info etc. Sending such info to your observability backend could get you in trouble. Have a look at https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation-api-incubator/src/main/java/io/opentelemetry/instrumentation/api/incubator/semconv/db/RedisCommandSanitizer.java The sanitizer doesn't really know what to do with the SEARCH command and removes all the arguments.

So my full command for starting the Spring Boot app looks like this:

-Dotel.instrumentation.common.default-enabled=true and -Dotel.instrumentation.lettuce.enabled=true are not needed as they are by default true

But some are missing such as db.query.text and db.query.parameter.. I'm not sure why db.statement is even appearing as a span attribute when it is open-telemetry/semantic-conventions#866.

Db client semantic conventions are currently undergoing an effort to declare them stable. This stabilization process will result in many changes to the existing conventions. To smoothen the transition to the new conventions the decision was not to implement the new conventions as the changes are made in the spec, but rather implement all changes together and have a flag that allows users to opt in for the new conventions or remain using the old ones.

[dasivon021]

Thank you for detailed clarification.

My command now looks like this:

java -javaagent:.\target\lib\opentelemetry-javaagent.jar -jar .\target\demo-0.0.1-SNAPSHOT.jar -Dotel.traces.exporter=otlp -Dotel.metrics.exporter=otlp -Dotel.logs.exporter=otlp -Dotel.instrumentation.lettuce.experimental-span-attributes=true -Dotel.instrumentation.lettuce.connection-telemetry.enabled=true -Dotel.instrumentation.common.db-statement-sanitizer.enabled=false -Dotel.service.name=demo-otel-redis -Dotel.exporter.endpoint=http://localhost:4317

Even with sanitization disabled I still see the same masked search statement with ?. I tried 2.3.0 & 2.4.0 javaagent.

[laurit]

When launching a java app whatever comes after -jar some.jar is an application argument that is passed to the main method. You'll have to move the -D flags before -jar for them to have any effect.

[dasivon021]

Thanks again for the correction on -jar command. Yes, I can confirm, the query is visible now:

"SEARCH myIdx @User\-Name:{John} TElNSVQ= 0 1 U09SVEJZ User-Name REVTQw=="

So the keywords TElNSVQ= -> LIMIT , U09SVEJZ -> SORTBY and REVTQw== -> DESC are Base64 encoded. Is there any tweak for this one? Wo would like to see exact query as in Redis monitor.

[laurit]

We don't have any tests with lettucemod so I can't test it out. Probably fixing this isn't too difficult. I'd guess that in

opentelemetry-java-instrumentation/instrumentation/lettuce/lettuce-5.1/library/src/main/java/io/lettuce/core/protocol/OtelCommandArgsUtil.java

Lines 37 to 45 in 8e10717

	if (argument instanceof KeyArgument) {
	KeyArgument keyArg = (KeyArgument) argument;
	return stringCodec.decodeKey(keyArg.codec.encodeKey(keyArg.key));
	}
	if (argument instanceof ValueArgument) {
	ValueArgument valueArg = (ValueArgument) argument;
	return stringCodec.decodeValue(valueArg.codec.encodeValue(valueArg.val));
	}
	return argument.toString();

we'd also need to have special handling for arguments of type ProtocolKeywordArgument