Span.currentSpan() returns a span whose Trace's id is 00000000000000000000000000000000 instead of the actual current trace's id

[yatamraghu7]

Hi All,

As provided a resolution here #2543
We are using mvn dependency approach for instrumenting our spring boot with opentelemetry.
We are getting the issue when we use Span currentSpan = Span.current();
PropagatedSpan{ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}}
Can you please tell us what can we change to get the proper traceId and spanId which was supposed to get instrumented.
We are not using agent in our project and also no java code has been written for instrumentation. Below is the bom that we are using

<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>io.opentelemetry.instrumentation</groupId>
				<artifactId>opentelemetry-instrumentation-bom</artifactId>
				<version>2.6.0</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
</dependencyManagement>

              <dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-sleuth</artifactId>
			<version>3.1.3</version>
			<exclusions>
				<exclusion>
					<groupId>org.springframework.cloud</groupId>
					<artifactId>spring-cloud-sleuth-brave</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-sleuth-otel-autoconfigure</artifactId>
			<version>1.1.2</version>
		</dependency>
                 <dependency>
			<groupId>io.opentelemetry</groupId>
			<artifactId>opentelemetry-exporter-otlp</artifactId>
		</dependency>
		<dependency>
			<groupId>io.opentelemetry.instrumentation</groupId>
			<artifactId>opentelemetry-spring-boot-starter</artifactId>
		</dependency>

Thanks

[laurit]

A trace/span id of all zeros denotes an invalid trace, this is used when there is no active trace. Without knowing anything about your application it is hard to tell why this happens. It would be best if you could provide a minimal application that reproduces your issue along with any necessary instructions. You may also try with the agent and see if it works any better.

[yatamraghu7]

Hi @laurit
Can you please let me know where can I find the agent compatible for my opentelemetry version that I am using which is 2.6.0
I would like to give a try with agent and if not working then I will provide a code snippet.

Thanks

[laurit]

Have a look at https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases

[yatamraghu7]

Hi @laurit

I tried using the java agent but still the same issue. Here is my problem in detail below.

Issue:

We are intercepting the our httpRequests using sleuth TracingClientHttpRequestInterceptor and below is the code snippet that I am using

@Configuration
public class SleuthConfiguration {

    @Autowired
    private CurrentTraceContext currentTraceContext;

    @Autowired
    private HttpClientHandler httpClientHandler;

    @Bean
    public RestTemplate restTemplate() {
        return new RestTemplateBuilder()
                .interceptors(TracingClientHttpRequestInterceptor.create(currentTraceContext, httpClientHandler))
            .build();
    }
}

As per my dependencies mentioned above I am excluding the brave trace propagators from sleuth and somehow our tracing interceptors are mixed up with opentelemetry. So we decided to get rid of sleuth and implement opentelemetry and when I am trying to achieve that I am having issue when trying to intercept our httpRequests using opentelemetry instead of sleuth which was provided above and below is the code snippet that I am using and dependencies

                 <dependency>
			<groupId>io.opentelemetry.instrumentation</groupId>
			<artifactId>opentelemetry-spring-boot-starter</artifactId>
			<version>2.7.0</version>
		</dependency>
		<dependency>
			<groupId>io.opentelemetry.instrumentation</groupId>
			<artifactId>opentelemetry-instrumentation-api</artifactId>
			<version>1.19.0</version>
		</dependency>

@Configuration
public class SleuthConfiguration {
    @Bean
    public RestTemplate restTemplate() {
        return new RestTemplateBuilder().additionalInterceptors(new RestTemplateInterceptor()).build();
    }
}

I am getting error that RestTemplateInterceptor cannot be resolved to a type as the dependecy was missing for this and if i am trying to add below dependency I am getting error

                <dependency>
			<groupId>io.opentelemetry.instrumentation</groupId>
			<artifactId>opentelemetry-resttemplate-1.5</artifactId>
			<version>1.19.0</version>
		</dependency>

How can i intercept propertly my httpRequests using opentelemetry ?

Thanks

[laurit]

Firstly sleuth and opentelemetry are competing tracing solutions. Sleuth questions are best asked from the sleuth community. If you decide to use open telemetry then a sample application would be handy. Currently it is quite unclear what you are attempting to do.

[yatamraghu7]

Hi @laurit

Here is a sample project and I wanted to see a traceId and spanId in my logs when two of my microservices are communicating with restTemplate.
https://github.com/yatamraghu7/opentelemetryInstrumentation/tree/main
Currently there are no traceId and spanid generated. Can you tell me where I was doing wrong. I have followed this
https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/spring/spring-web/spring-web-3.1/library

I wanted to do manual instrumentation with opentelemetry because the opentelemetry java agent jar will support RestTemplate instrumentation but not the WebClinet and WebSocket communication between my microservices which I already have currently in my main project.
So it would be great if you can point me to a proper documentation where I can get the information of manual instrumentation of opentelemetry for RestTemplate, WebClient and WebSocket communications.

Thanks

[yatamraghu7]

Hi @laurit @jeanbisutti @zeitlinger

Sorry that my question was little bit confusing as I have provided a log statement from my sample application which is having logexporter in the log. I meant I would like to have my log formatted to the below format

2024-09-13 16:37:18.030 INFO [Consumer,traceId=01011111001110102222222222222222,spanId=c21a735037e2eb4d]

As the current format looks like below

2024-09-13 16:35:46.725 INFO 75976 --- [nio-8080-exec-1] : 'GET' : 01011111001110102222222222222222 5fa635b7db8ac85e CLIENT [tracer: io.opentelemetry.spring-web-3.1:2.7.0-alpha] AttributesMap{data={http.request.method=GET, server.address=localhost, http.response.status_code=200, url.full=http://localhost:8081/hello/fromRestTemplate, server.port=8081}, capacity=128, totalAddedValues=5}

Just ignore the exporter statement in the log. What I am asking is can i somehow get the traceId and spanId appended to each and every line of the log that my logger is writing ?

I would like to achieve this with may be spring boot application property as mentioned below which is not a working example but

logging.pattern.level=%5p [${spring.application.name:},traceId=%X{traceId:-},spanId=%X{spanId:-}] %d{yyyy-MM-dd HH:mm:ss.SSS} %logger{36} - %msg%n

Kindly let me know if this can be achieved without making a logback.xml or log4j.xml

Please checkout my sample code at below repo
https://github.com/yatamraghu7/ManualInstrumentationOtel

Thanks

[yatamraghu7]

Hi @jkwatson

I can see that you have suggested some pom changes here #3074 and it worked for him to get the traceId and spanId
I am also facing the same issue here that my traceId and spanId that was propagated by opentelemetry was not injected to MDC.
Please check my code at https://github.com/yatamraghu7/ManualInstrumentationOtel
In my logs below I am getting null value can you please suggest where is the mistake happening in my application

You can see in my above comments I wanted to achieve something similar to what was mentioned https://github.com/open-telemetry/opentelemetry-java-instrumentation/discussions/3074

Thanks

[yatamraghu7]

Hi @laurit and @jkwatson

I have added the logback.xml file and also mvn dependency below

                <dependency>
			<groupId>io.opentelemetry.instrumentation</groupId>
			<artifactId>opentelemetry-logback-mdc-1.0</artifactId>
			<version>2.7.0-alpha</version>
		</dependency>

I am still not able to understand why the traceId and spanId are not getting included in MDC from opentelemetry ?

As you can see below screen shot I am missing the traceId and spanId from MDC and also the response from SERVER side also not appending the traceId and spanId to the log statement

2024-09-16 15:56:14 - c.m.c.C.C.ConsumerController - INFO  - The retrieved MDC traceId and spanId will be [null] and [null]
				trace_id:01011111001110102222222222222222 span_id:65be5262ffc3f3b3 
2024-09-16 15:56:14 - c.m.c.C.C.ConsumerController - INFO  - This is a call going to happen from consume method to producer class controller method i.e. sayHelloUsingWebClient
				trace_id:01011111001110102222222222222222 span_id:65be5262ffc3f3b3 
2024-09-16 15:56:14 - i.o.e.logging.LoggingSpanExporter - INFO  - 'GET' : 01011111001110102222222222222222 d2d44619fd5ecaa8 CLIENT [tracer: io.opentelemetry.spring-webflux-5.3:2.7.0-alpha] AttributesMap{data={server.address=localhost, server.port=8081, http.request.method=GET, http.response.status_code=200, url.full=http://localhost:8081/hello/fromWebClient}, capacity=128, totalAddedValues=5}
				trace_id:01011111001110102222222222222222 span_id:d2d44619fd5ecaa8 
2024-09-16 15:56:14 - i.o.e.logging.LoggingSpanExporter - INFO  - 'GET /consume/usingWebClient' : 01011111001110102222222222222222 65be5262ffc3f3b3 SERVER [tracer: io.opentelemetry.spring-webmvc-5.3:2.7.0-alpha] AttributesMap{data={network.peer.address=127.0.0.1, server.address=localhost, client.address=127.0.0.1, user_agent.original=curl/7.88.1, url.path=/consume/usingWebClient, server.port=8080, network.peer.port=50532, http.request.method=GET, http.response.status_code=200, http.route=/consume/usingWebClient, network.protocol.version=1.1, url.scheme=http}, capacity=128, totalAddedValues=12}
				trace_id: span_id: 

Can someone please tell me why this is happening.
Sample code can be checked at here https://github.com/yatamraghu7/ManualInstrumentationOtel

Thanks

[laurit]

As you can see below screen shot I am missing the traceId and spanId from MDC

You won't be able to get the trace id with MDC.get("trace_id"). Have a look at the instrumentation in https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/logback/logback-mdc-1.0/library You'll see that the trace id isn't really added to the MDC but to the copy of MDC that is attached to the logging event.

and also the response from SERVER side also not appending the traceId and spanId to the log statement

I have no idea what you mean by that. If you mean that it is missing from stuff logged by LoggingSpanExporter then that is expected.
Overall to me it seems that trace_id and span_id are added when you log from your own code, so seems to be working fine.

[yatamraghu7]

Hi @laurit

Is there any way that I can customize the pattern of the logback.xml file like below

Current:

%d{HH:mm:ss.SSS} trace_id=%X{trace_id} span_id=%X{span_id} trace_flags=%X{trace_flags} %msg%n

Wanted to Have like:

%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] [%X{trace_id}, %X{span_id}] %-5level %logger{36} - %msg%n

Is this even possible that we can customize the logging pattern as we like ?

Thanks

[laurit]

Could you elaborate what is preventing you from changing the pattern in logback.xml? Whatever it currently has was added by you.

[yatamraghu7]

Hi @laurit

It is working fine and I am able to set the custom logging pattern.

Thanks

[yatamraghu7]

I got the resolution from @laurit and thanks for support.