instrumentation does not work during deployment

[abaybektursun]

I am trying to get OpenTelemtry working with my app (For debugging LLMs). It works when I run it locally, however it stops working after deploying with SST v3 ("sst": "3.2.12")
I have this in my next.config.js

module.exports = {
  experimental: {
    instrumentationHook: true
  }
}

instrumentation.ts

// Ensure this is in the root of your project, not in app or pages directories
export async function register() {
  if (process.env.NEXT_RUNTIME === 'nodejs') {
    await import('./instrumentation.node')
  }
}

instrumentation.node.js

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-node';
import { diag, DiagConsoleLogger, DiagLogLevel } from '@opentelemetry/api';

// Enable //console.logging for OpenTelemetry
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);

console.log('Starting OpenTelemetry initialization');

const isProduction = process.env.NODE_ENV === 'production';
console.log(`(instrumentation.node.js) Environment: ${isProduction ? 'Production' : 'Development'}`);

const traceExporter = new OTLPTraceExporter({
  url: '.......',
});

//console.log(`Trace exporter URL: ${traceExporter.url}`);

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'tracing',
    [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: isProduction ? 'production' : 'development',
  }),
  spanProcessors: [new SimpleSpanProcessor(traceExporter)],
});

// Start the SDK without assuming it returns a Promise
try {
  sdk.start();
  console.log('OpenTelemetry SDK started successfully');
} catch (error) {
  console.error('Error starting OpenTelemetry SDK:', error);
}

process.on('SIGTERM', () => {
  sdk.shutdown()
    .then(() => console.log('Tracing terminated'))
    .catch((error) => console.log('Error terminating tracing', error))
    .finally(() => process.exit(0));
});

console.log('OpenTelemetry setup script finished');

This entire file seems to be ignored since there are no logs from instrumentation.node.js in CloudWatch or SST.

[conico974]

Have you tested the solution in #520 ? It uses @vercel/otel instead

My guess is that one of the import is failing for some reason. Does it work with the standalone output ?

And BTW this cannot properly work on lambda, you're not awaiting things properly here, the last exported trace of a request will likely never get sent.

[oldbettie]

The solution in my post did not work I spent about a week trying different methods and conico was helpful but everything I tried resulted in the same failure. We have switched to railway for deployment until opennext supports Otel

[conico974]

The solution in the other post did work for me with @vercel/otel.
If it still doesn't work, you'll need to provide a reproduction

[pondorasti]

Running into the same issue. I won't be able to immediately provide a reproduction. Same project is currently deployed on Vercel, so I would expect for things to just work out of the box.

In the meantime, I'm going to drop my build error here in case it sparks some ideas:

11:14:01.939 | ✘ [ERROR] Could not resolve "../platform"
-- | --
11:14:01.939 |  
11:14:01.939 | .next/standalone/node_modules/.pnpm/@[email protected]/node_modules/@opentelemetry/api/build/src/internal/global-utils.js:19:27:
11:14:01.939 | 19 │ const platform_1 = require("../platform");

[conico974]

Very likely related to this vercel/next.js#68740