Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ReactiveMessaging "CDI container is not available" #26342

Closed
AdisNezirovic opened this issue Sep 20, 2023 · 9 comments
Closed

ReactiveMessaging "CDI container is not available" #26342

AdisNezirovic opened this issue Sep 20, 2023 · 9 comments
Assignees
@benjamin-confino
Labels
in:MicroProfile/Reactive release bug This bug is present in a released version of Open Liberty release:24001

Comments

@AdisNezirovic
Copy link

AdisNezirovic commented Sep 20, 2023 •
edited
Loading

Describe the bug

When trying to inject RestClient into Bean which contains reactive messaging channel, I get "CDI container is not available", and application fails on startup.

[INFO] [WARNING ] CDI container is not available
[INFO] [ERROR   ] SRMSG00200: The method net.example.channels.KafkaEvents#consume has thrown an exception
[INFO] SRMSG00103: Exception thrown when calling the method net.example.channels.KafkaEvents#consume
[INFO] [ERROR   ] SRMSG00201: Error caught while processing a message in method net.example.channels.KafkaEvents#consume

The error seems to be related to OpenTelemetry feature (see trace bellow), but I'd really like to avoid removing that totally from the app, since we use it heavily.

Steps to Reproduce

From OpenLiberty starter template with MicroProfile 6.0, and following sample classes:

@RegisterRestClient(baseUri = "http://localhost:8080")
public interface ApiClient {
    @POST
    @Path("/")
    @Consumes(MediaType.APPLICATION_JSON)
    public CompletionStage<Response> postEvent(Event ev);
}

@ApplicationScoped
public class KafkaEvents {
    @Inject @RestClient ApiClient apiClient;

    @Incoming("in")
    public Uni<Void> consume(Message<List<Event>> stream) {
        return Uni.createFrom().voidItem();
    }
}

The channel is configured for Kafka in microprofile-config.properties

mp.messaging.incoming.in.bootstrap.servers=...
mp.messaging.incoming.in.connector=smallrye-kafka
mp.messaging.incoming.in.topic=...
mp.messaging.incoming.in.group.id=openliberty-bug-test
mp.messaging.incoming.in.auto.offset.reset=earliest
...

Expected behavior

Application starting and being able to use RestClient from within consume() method (as you can see we haven't even come to that).
My current workaround is to use Java 11 async http client, which is slightly too involved for my taste.

Diagnostic information:

  • OpenLiberty Version: 23.0.0.9 (microProfile-6.0 feature)
  • SmallRye: 4.9.0
  • Java Version: openjdk version "17.0.8.1" 2023-08-24

Trace:

[9/20/23, 19:33:53:701 CEST] 0000004f io.smallrye.reactive.messaging.provider                      E SRMSG00201: Error caught while processing a message in method net.example.channels.KafkaEvents#consume
io.smallrye.reactive.messaging.providers.ProcessingException: SRMSG00103: Exception thrown when calling the method net.example.channels.KafkaEvents#consume
   at io.smallrye.reactive.messaging.providers.AbstractMediator.lambda$initialize$7(AbstractMediator.java:127)
   at io.smallrye.reactive.messaging.providers.AbstractMediator.invoke(AbstractMediator.java:141)
   at io.smallrye.reactive.messaging.providers.AbstractMediator.lambda$invokeOnMessageContext$8(AbstractMediator.java:149)
   at io.smallrye.reactive.messaging.providers.locals.LocalContextMetadata.lambda$invokeOnMessageContext$0(LocalContextMetadata.java:34)
   at io.smallrye.reactive.messaging.providers.locals.LocalContextMetadata.lambda$invokeOnMessageContext$1(LocalContextMetadata.java:53)
   at io.smallrye.reactive.messaging.providers.helpers.VertxContext.runOnContext(VertxContext.java:24)
   at io.smallrye.reactive.messaging.providers.locals.LocalContextMetadata.lambda$invokeOnMessageContext$2(LocalContextMetadata.java:51)
   at io.smallrye.mutiny.operators.uni.builders.UniCreateWithEmitter.subscribe(UniCreateWithEmitter.java:22)
   at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
   at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
   at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
   at io.smallrye.mutiny.operators.uni.UniOnItemOrFailureFlatMap.subscribe(UniOnItemOrFailureFlatMap.java:27)
   at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
   at io.smallrye.mutiny.converters.uni.UniToMultiPublisher$UniToMultiSubscription.request(UniToMultiPublisher.java:73)
   at io.smallrye.mutiny.subscription.SwitchableSubscriptionSubscriber.setOrSwitchUpstream(SwitchableSubscriptionSubscriber.java:205)
   at io.smallrye.mutiny.subscription.SwitchableSubscriptionSubscriber.onSubscribe(SwitchableSubscriptionSubscriber.java:107)
   at io.smallrye.mutiny.converters.uni.UniToMultiPublisher.subscribe(UniToMultiPublisher.java:25)
   at io.smallrye.mutiny.groups.MultiCreate$1.subscribe(MultiCreate.java:165)
   at io.smallrye.mutiny.operators.multi.MultiConcatMapOp$ConcatMapMainSubscriber.onItem(MultiConcatMapOp.java:144)
   at io.smallrye.mutiny.operators.multi.MultiMapOp$MapProcessor.onItem(MultiMapOp.java:50)
   at io.smallrye.mutiny.operators.multi.MultiOperatorProcessor.onItem(MultiOperatorProcessor.java:99)
   at io.smallrye.reactive.messaging.providers.locals.ContextOperator$ContextMulti$ContextProcessor.lambda$onItem$1(ContextOperator.java:71)
   at io.smallrye.reactive.messaging.providers.helpers.VertxContext.lambda$runOnContext$0(VertxContext.java:26)
   at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
   at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:246)
   at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:43)
   at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
   at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
   at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
   at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
   at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
   at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.reflect.InvocationTargetException
   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.base/java.lang.reflect.Method.invoke(Method.java:568)
   at io.smallrye.reactive.messaging.providers.AbstractMediator.lambda$initialize$7(AbstractMediator.java:125)
   ... 33 more
Caused by: java.lang.IllegalStateException: Could not find deployment
   at com.ibm.ws.cdi.impl.AbstractCDIRuntime.getCDI(AbstractCDIRuntime.java:163)
   at jakarta.enterprise.inject.spi.CDI.getCDIProvider(CDI.java:78)
   at jakarta.enterprise.inject.spi.CDI.current(CDI.java:65)
   at io.openliberty.microprofile.telemetry.internal.rest.TelemetryClientFilter.getCurrent(TelemetryClientFilter.java:65)
   at io.openliberty.microprofile.telemetry.internal.rest.TelemetryRestClientBuilderListener.onNewBuilder(TelemetryRestClientBuilderListener.java:23)
   at org.eclipse.microprofile.rest.client.RestClientBuilder.lambda$newBuilder$0(RestClientBuilder.java:57)
   at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
   at org.eclipse.microprofile.rest.client.RestClientBuilder.newBuilder(RestClientBuilder.java:55)
   at org.jboss.resteasy.microprofile.client.RestClientDelegateBean.create(RestClientDelegateBean.java:122)
   at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:64)
   at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
   at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
   at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:680)
   at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:780)
   at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92)
   at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:345)
   at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:356)
   at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:69)
   at com.ibm.ws.cdi.impl.weld.injection.WebSphereInjectionServicesImpl$3.run(WebSphereInjectionServicesImpl.java:325)
   at com.ibm.ws.cdi.impl.weld.injection.WebSphereInjectionServicesImpl$3.run(WebSphereInjectionServicesImpl.java:322)
   at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
   at com.ibm.ws.cdi.impl.weld.injection.WebSphereInjectionServicesImpl.aroundInject(WebSphereInjectionServicesImpl.java:322)
   at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:46)
   at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:71)
   at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117)
   at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:161)
   at org.jboss.weld.contexts.AbstractContext.get(AbstractContext.java:96)
   at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
   at org.jboss.weld.bean.ContextualInstanceStrategy$ApplicationScopedContextualInstanceStrategy.get(ContextualInstanceStrategy.java:140)
   at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
   at org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:102)
   at org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:131)
   at net.example.channels.KafkaEvents$Proxy$_$$_WeldClientProxy.consume(Unknown Source)
   ... 38 more
@AdisNezirovic AdisNezirovic added the release bug This bug is present in a released version of Open Liberty label Sep 20, 2023
@AdisNezirovic AdisNezirovic changed the title RectiveMessaging "CDI container is not available" ReactiveMessaging "CDI container is not available" Sep 20, 2023
@benjamin-confino
Copy link
Member

benjamin-confino commented Sep 21, 2023 •
edited
Loading

CDI depends on application metadata held in the Thread to know which application a request is coming from. The IllegalStateException from AbstractCDIRuntime is telling me that this metadata is absent, which usually happens when a new thread is created without the state being transferred over from its parent.

Am I correct in assuming you are bundling the smallrye library directly into your application? I see that we have code to set that state when creating a new thread in the mpReactiveStreams-3.0 feature. Currently this is only shipped in nightly builds (it is targeted for 230011-beta) so I would not recommend using it in production. However you could give it a try to see if it works locally.

I'm afraid there is no workaround to use OpenTelemerty without CDI. OpenTelmetry is depending on CDI to tell it which application the current Thread is working in, if it didn't use CDI it would have to aquire this information itself and it would do so using the exact same MetaData.

In the meanwhile I will write a FAT test based on your code.

@AdisNezirovic
Copy link
Author

AdisNezirovic commented Sep 21, 2023 •
edited
Loading

Yes, you are correct about bundling, here is extract from my pom.xml

<dependency>
    <groupId>io.smallrye.reactive</groupId>
    <artifactId>smallrye-reactive-messaging-kafka</artifactId>
    <version>4.9.0</version>
</dependency>
<dependency>
    <groupId>io.smallrye.reactive</groupId>
    <artifactId>mutiny-reactive-streams-operators</artifactId>
    <version>2.4.0</version>
</dependency>

I've tried the latest nightly and it still complains loudly, but it actually seems to work (I'm able to consume the messages and use RestClient from within the channel)

Log:

[INFO] [WARNING ] CDI container is not available
[INFO] [ERROR   ] CWMOT5002E: An internal error occurred in MicroProfile Telemetry. The error is: java.lang.NullPointerException: Cannot invoke "com.ibm.ws.runtime.metadata.ComponentMetaData.getJ2EEName()" because the return value of "com.ibm.ws.threadContext.ComponentMetaDataAccessorImpl.getComponentMetaData()" is null.

Trace:

[9/21/23, 14:37:35:219 CEST] 00000050 com.ibm.ws.logging.internal.impl.IncidentImpl                I FFDC
1015I: An FFDC Incident has been created: "java.lang.NullPointerException: Cannot invoke "com.ibm.ws.runt
ime.metadata.ComponentMetaData.getJ2EEName()" because the return value of "com.ibm.ws.threadContext.Compo
nentMetaDataAccessorImpl.getComponentMetaData()" is null io.openliberty.microprofile.telemetry10.internal
.rest.TelemetryRestClientBuilderListener 33" at ffdc_23.09.21_14.37.35.0.log
[9/21/23, 14:37:35:221 CEST] 00000050 telemetry10.internal.rest.TelemetryRestClientBuilderListener E CWMOT5002E: An internal error occurred in MicroProfile Telemetry. The error is: java.lang.NullPointerException: Cannot invoke "com.ibm.ws.runtime.metadata.ComponentMetaData.getJ2EEName()" because the return value of "com.ibm.ws.threadContext.ComponentMetaDataAccessorImpl.getComponentMetaData()" is null.
[9/21/23, 14:37:35:502 CEST] 00000050 org.jboss.resteasy.microprofile.client.RestClientBuilderImpl W CDI container is not available

Please let me know if I can provide more info.

@benjamin-confino
Copy link
Member

benjamin-confino commented Sep 21, 2023 •
edited
Loading

I have a suspicion that's a known bug where the fix is delivered but hasn't made it into the nightly builds. To help me confirm please could you provide the file ffdc_23.09.21_14.37.35.0.log and any other files in the ffdc directory.

Thank you.

@benjamin-confino benjamin-confino mentioned this issue Sep 22, 2023
Merged
@AdisNezirovic
Copy link
Author

Requested file attached:

ffdc_23.09.21_14.37.35.0.log

@benjamin-confino
Copy link
Member

Unfortunately that is not the issue I previously dealt with. Looking at your FFDC there is nothing in that stack which will register a J2EEName onto the thread context. And since you get this error its safe to assume there's nothing in the previous thread propagating it.

How are you spawning new threads? Using the ManagedExecutorService provided by our concurrent feature is a safe option.

@benjamin-confino
Copy link
Member

Hello again.

It has been a while since you replied, I hope this means all is well on your end?

When an issue is over I like to create a test to ensure Liberty doesn't regress, would it be possible for you to share your code or a dummy version which I can use to base the test on?

@AdisNezirovic
Copy link
Author

Hello, real life interferes, please leave this open for a few days more so I can create another repeatable text with an executor (since that was the whole idea).

@benjamin-confino
Copy link
Member

Real life happens to us all, see you in a few days more.

@AdisNezirovic
Copy link
Author

Hello,

I just want to report that everything seems to work as expected in the current beta (23.0.0.11-beta), e.g. something simple like reading payloads from Kafka and using (blocking) Rest client from within the channel consumer.

This time I'm using mpReactiveMessaging-3.0 feature, no explicit/custom smallrye dependencies. Hope this will soon be released in stable version!

@ApplicationScoped
public class KafkaEvents {
    @Inject @RestClient ApiClient apiClient;

    @Resource(lookup = "java:comp/DefaultManagedExecutorService")
    ManagedExecutorService execService;

    @Incoming("in")
    public CompletionStage<Void> consume(String msg) {
        execService.execute(
                () -> {
                    // let's pray^^pretend only 2xx/3xx will be returned
                    Response r = apiClient.postEvent(msg);
                    System.out.println(r.getStatusInfo().getReasonPhrase());
                });

        return CompletableFuture.completedStage(null);
    }
}

Thank you for the help and support, you can close the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benjamin-confino benjamin-confino

Labels
in:MicroProfile/Reactive release bug This bug is present in a released version of Open Liberty release:24001
Projects
MicroProfile UK
Status: Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@benjamin-confino @abutch3r @LibbyBot @AdisNezirovic