Auto Instrumentation of AWS java lambda

[vinitaaher]

I am doing auto instrumentation for AWS java lambda. I have created one AWS lambda and accessing this lambda in myapp using API Gateway.When I see traces in jaeger , I am able to see only one span for http url only. But I want see 2 span there ,one for HTTP req. and another for lambda function.

[anuraaga]

Hi @vinitaaher - can you describe more precisely what you're trying? I see a java command there, is that the app that is creating an HTTP request to API gateway?

We see the HTTP client span corresponding to what your application sent, this is expected. There is no way you would be able to see lambda spans in such a setup because Jaeger is running on your localhost, but lambda is running in cloud infrastructure and does not have access to it. To be able to see everything together, you will generally need to be running Jaeger in your infrastructure, and configure the Java agent to use the Jaeger exporter and point to it.

[anuraaga]

Auto instrumentation of Java lambda is not officially supported, however it does sort of work. You need to create a layer with the agent in it and use JAVA_TOOL_OPTIONS to activate the agent in your lambda function. Unfortunately, coldstart time goes up a lot so this only makes sense with provisioned concurrency. We don't currently have documentation on this though since the experience isn't great.

You may want to give the lambda wrappers a try - they only instrument the function, not any client invocations you my make, but could possibly help.

https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/master/instrumentation/aws-lambda-1.0/library#using-wrappers

[vinitaaher]

@anuraaga - Thank You !

[vinitaaher]

Hi @anuraaga - As per your suggestion I have added agent by using JAVA_TOOL_OPTIONS. Please refer :
Following is the simple lambda function code.

public class LambdaApi implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent>{
    @Override
    public APIGatewayProxyResponseEvent handleRequest(APIGatewayProxyRequestEvent input,
        Context context) {
        LambdaLogger logger = context.getLogger();
        Map<String, String> headers = new HashMap<>();
        headers.put("Content-Type", "application/json");
        headers.put("X-Custom-Header", "application/json");
        String output = "Welcome !!!";
        logger.log(output);
        APIGatewayProxyResponseEvent response = new APIGatewayProxyResponseEvent()
            .withHeaders(headers)
            .withStatusCode(200)
            .withBody(output);
        return response;
    }
}

Environment variable that I am passing to AWS Lambda :

       Key                                               Value
JAVA_TOOL_OPTIONS                      -javaagent:lib/opentelemetry-javaagent-all.jar

OTEL_EXPORTER                           jaeger

OTEL_EXPORTER_JAEGER_ENDPOINT           192.168.9.151:14268/api/traces

OTEL_EXPORTER_JAEGER_SERVICE_NAME       MyApp

Following is the log for AWS Lambda :

START RequestId: 0e98f1ca-40d8-42c8-b1f7-99cd1e06a54b Version: $LATEST
Picked up JAVA_TOOL_OPTIONS: -javaagent:lib/opentelemetry-javaagent-all.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[opentelemetry.auto.trace 2021-01-19 12:22:14:348 +0000] [main] INFO io.opentelemetry.javaagent.tooling.TracerInstaller - Installed span exporter: io.opentelemetry.exporter.jaeger.JaegerGrpcSpanExporter
[opentelemetry.auto.trace 2021-01-19 12:22:14:649 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 0.12.0
EXTENSION	Name: cloudwatch_lambda_agent	State: Ready	Events: [INVOKE,SHUTDOWN]
Welcome !!!END RequestId: 0e98f1ca-40d8-42c8-b1f7-99cd1e06a54b
REPORT RequestId: 0e98f1ca-40d8-42c8-b1f7-99cd1e06a54b	Duration: 38803.10 ms	Billed Duration: 38804 ms	Memory Size: 512 MB	Max Memory Used: 160 MB	
XRAY TraceId: 1-6006ced3-3c8f8fdc34952dda351a34a7	SegmentId: 14c3a43d5a63fc85	Sampled: true	

I am able to see in log that is io.opentelemetry.javaagent.tooling.TracerInstaller - Installed span exporter: io.opentelemetry.exporter.jaeger.JaegerGrpcSpanExporter. but not getting any traces in jaeger yet. Is there missing anything?Can you please suggest what I am missing here.
I also tried using aws-opentelemetry-agent.jar as java agent , but facing same issue.

[anuraaga]

Hi @vinitaaher - do you mind setting OTEL_EXPORTER=logging to see if any traces are being generated? If they are it would be an issue with connectivity with Jaeger, if not an instrumentation one, either way we'll know better what to examine.

[vinitaaher]

@anuraaga - Thanks, I did OTEL_EXPORTER=logging then it is working. i.e It is issue with connectivity to Jaeger.