[Regression] New HTTP/2 Warnings

[lprimak]

Brief Summary

New warnings appear below. There were no warnings before.
These warnings started appearing with Payara 6.2024.3

[#|2024-03-14T16:09:23.862+0000|WARNING|Payara 6.2024.3|org.glassfish.grizzly.http2.frames.SettingsFrame|_ThreadID=91;_ThreadName=http-thread-pool::http-listener-2(3);_TimeMillis=1710432563862;_LevelValue=900;|
  Setting 2,794 is unknown and will be ignored|#]

[#|2024-03-14T16:09:32.167+0000|WARNING|Payara 6.2024.3|org.glassfish.grizzly.http2.frames.SettingsFrame|_ThreadID=90;_ThreadName=http-thread-pool::http-listener-2(2);_TimeMillis=1710432572167;_LevelValue=900;|
  Setting 6,714 is unknown and will be ignored|#]

Expected Outcome

No warnings

Current Outcome

Warnings

Reproducer

No simple reproducer (at least not yet)

Operating System

Any

JDK Version

Any

Payara Distribution

Payara Server Full Profile

[Elifzeynepedman]

Hi @lprimak,

Could you please provide a simple-to-follow scenario on how to reproduce this on the latest version? A reproducer should ideally follow the SSCCE rules: http://www.sscce.org/. It will greatly help us to find the cause and fix it.

Thank you,
Elif

[lprimak]

I’m sorry I don’t have one. I have production monitoring logs and this was a new alert since upgrading

[Elifzeynepedman]

Hi @lprimak,

Could you please provide answers to the following questions, it would help us proceed with the issue;

1. Can you describe the specific circumstances under which the HTTP/2 warning appears?
2. Are you encountering the warning while using an HTTP/2 enabled endpoint in your application
3. Could you specify if you're utilizing a particular development framework that may interact with HTTP/2.

It would be great if you could provide as much information as you could for us to get a better understanding.

Thank you,
Elif

[lprimak]

1. I can't tell when this specifically happens, it's random / sporadic.
2. Yes, http/2 endpoint is enabled (by default) and that's what I am using.
3. JSF and PrimeFaces (latest release version)

I am sorry I can't be more helpful but the nature of this problem prevents this unfortunately.

[github-actions]

Greetings,
It's been more than 5 days since we requested more information or an update from you on the details of this issue. Could you provide an update soon, please?
We're afraid that if we do not receive an update, we'll have to close this issue due to inactivity.

[lprimak]

Not abandoned :)

[lprimak]

I found a way to consistently reproduce the issue. Stay tuned.

[lprimak]

Didn't work

Ok, here's how to reproduce. It's not quite trivial but it's simple and self-contained:

I did it in a docker container, but you don't have to.
You can run any JDK starting with 17 and higher

Run docker container:

docker run -it --rm azul/zulu-openjdk-alpine:17-latest

Set up docker container:

apk update && apk add git maven curl firefox

Clone and run tests:

git clone https://github.com/flowlogix/flowlogix.git
cd flowlogix
curl https://raw.githubusercontent.com/lprimak/infra/main/scripts/cloud/docker/_builders/agent-maven-settings.xml -o /flowlogix/settings.xml
mvn verify -ntp -gs /flowlogix/settings.xml -Pui-test -Dwebdriver.browser=firefox -Dtest=none -Dsurefire.failIfNoSpecifiedTests=false

---

See the output in the logs:

less /flowlogix/target/dependency/payara6/glassfish/domains/domain1/logs/server.log

You will see the errors in the log file

[lprimak]

Well, spoke too soon, the reproducer above fails to reproduce the issue. It's still not pinned down when / how this happens but it does.

[lprimak]

I guess the only thing I learned today is that it only happens when Chrome is the client

[Elifzeynepedman]

Hi @lprimak,

Thank you for your efforts in reporting this issue. Unfortunately, as we have not received the necessary reproducer to consistently replicate the problem, we are unable to proceed with further investigation. Therefore, we will have to close this issue. If you are able to provide the details needed in the future, please feel free to reopen the issue or submit a new one.

Thank you,
Elif

[lprimak]

Hi, Elif,

Please reopen this.
Just because it's not "reliably reproducible" at the moment doesn't mean the issue doesn't exist.
I have worked hard to reproduce this issue, and even if unsuccessful, I don't want to see all that work wasted.
Thank you

[phillipross]

An advantage to keeping things like this open in the issue tracker can help provide visibility to community users who may run into the issue and have more resources/bandwidth to provide higher quality reproducers that help find the true root cause and and/or fix underlying issues.

Specifically, issues related to http/2 have been notoriously difficult to troubleshoot, fix, detect regressions in, or even make sense of or differentiate between new issues, regressions, or somewhere in between 😬

[fturizo]

@lprimak, @phillipross, we understand your comments. We'd have no problem keeping this issue open until a reliable reproducer is created to properly fix the issue. Still, we have a clear support policy that states that all bug reports must be accompanied by a reliable reproducer to be formally escalated to our Platform engineers, so keep this in mind.

However, in the spirit of conceding that HTTP/2 issues are indeed challenging to troubleshoot, we'll keep this issue open for another two months to give you (or other community users) the opportunity to share a reliable (or semi-reliable) reproducer to help us patch the issue. I'm afraid that if no reproducer is found, we'll proceed to close the issue again.

[roberto-cisternino]

I have similar error along with others but not related to my code.

[2024-06-25T19:00:39.324+0000] [Payara 6.2023.10] [SEVERE] [AS-WEB-CORE-00037] [jakarta.enterprise.web.core] [tid: _ThreadID=36 _ThreadName=http-thread-pool::http-listener-1(4)] [timeMillis: 1719342039324] [levelValue: 1000] [[
An exception or error occurred in the container during the request processing
java.net.ProtocolException: Host header not found in request
at com.sun.web.security.RealmAdapter.getHostAndPort(RealmAdapter.java:1126)
at com.sun.web.security.RealmAdapter.redirect(RealmAdapter.java:1096)
at com.sun.web.security.RealmAdapter.hasUserDataPermission(RealmAdapter.java:384)
at com.sun.web.security.RealmAdapter.hasUserDataPermission(RealmAdapter.java:326)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:451)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:726)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:372)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:239)
at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:174)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:153)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:196)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515)
at java.base/java.lang.Thread.run(Thread.java:840)
]]

[2024-06-25T19:13:40.879+0000] [Payara 6.2023.10] [WARNING] [] [org.glassfish.grizzly.http2.frames.SettingsFrame] [tid: _ThreadID=45 _ThreadName=http-thread-pool::http-listener-2(5)] [timeMillis: 1719342820879] [levelValue: 900] [[
Setting 8 is unknown and will be ignored]]

[2024-06-25T19:13:46.985+0000] [Payara 6.2023.10] [WARNING] [] [org.glassfish.grizzly.http2.frames.SettingsFrame] [tid: _ThreadID=41 _ThreadName=http-thread-pool::http-listener-2(1)] [timeMillis: 1719342826985] [levelValue: 900] [[
Setting 8 is unknown and will be ignored]]

[2024-06-25T19:40:34.187+0000] [Payara 6.2023.10] [WARN] [] [org.jboss.weld.Servlet] [tid: _ThreadID=36 _ThreadName=http-thread-pool::http-listener-1(4)] [timeMillis: 1719344434187] [levelValue: 900] [[
WELD-000714: HttpContextLifecycle guard leak detected. The Servlet container is not fully compliant. The value was 1]]

[2024-06-25T19:40:34.193+0000] [Payara 6.2023.10] [WARN] [] [org.jboss.weld.Context] [tid: _ThreadID=36 _ThreadName=http-thread-pool::http-listener-1(4)] [timeMillis: 1719344434193] [levelValue: 900] [[
WELD-000225: Bean store leak detected during org.jboss.weld.module.web.context.http.HttpRequestContextImpl association: org.apache.catalina.connector.RequestFacade@4379c2b5]]

[fturizo]

@roberto-cisternino, would you be able to share a self-contained reproducer that can help us identify how these warnings are triggered?

[lprimak]

I actually got the same errors in Payara 5 so this is a long standing issue not a regression.