dinky.log:Protocol message end-group tag did not match expected tag

[rollerQ]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

1.Environment：
dinky 0.7.2
flink1.12(hw-mrs)：Because Huawei Cloud flink cannot be upgraded, only a lower version of dinky can be used

2.Flink yarn per job task:
use example sql 【hello word】：https://cloud.tencent.com/developer/article/1925451：五、Yarn-Per-Job 实践

3.Configuration：

4.issue: The task can run normally on yarn, but dinky continues to circle on the submit page and stops after a time out. The following error is displayed in dinky's log

I've tried all kinds of packages：
flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar、
flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
flink-shaded-hadoop-2-uber-3.0.0-cdh6.2.0-7.0.jar.tar.gz
flink-shaded-hadoop2-uber-1.7.0.0101-hw-ei-12.jar

What you expected to happen

dinky can feedback the submitted task status normally. The operation and maintenance center displays the task that has just been submitted, but it does not

How to reproduce

flink test connected to Huawei cloud may be required

Anything else

no

Version

0.7.2

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[rollerQ]

add log：
dlink] 2024-02-02 14:10:46 CST INFO org.apache.flink.yarn.YarnClusterDescriptor 206 getLocalFlinkDistPath - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
[dlink] 2024-02-02 14:10:46 CST WARN org.apache.flink.yarn.YarnClusterDescriptor 333 isReadyForDeployment - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.flink.yarn.YarnClusterDescriptor 643 logIfComponentMemNotIntegerMultipleOfYarnMinAllocation - The configured JobManager memory is 768 MB. YARN will allocate 1024 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 256 MB may not be used by Flink.
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.flink.yarn.YarnClusterDescriptor 550 deployInternal - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1024, slotsPerTaskManager=1}
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.flink.runtime.util.config.memory.MemoryBackwardsCompatibilityUtils 56 lambda$getConfWithLegacyHeapSizeMappedToNewConfigOption$0 - 'jobmanager.memory.process.size' is not specified, use the configured deprecated task manager heap value (1024.000mb (1073741824 bytes)) for it.
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils 282 capToMinMax - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.flink.yarn.YarnClusterDescriptor 1058 startAppMaster - Submitting application master application_1670294478436_4074955
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.hadoop.io.retry.RetryInvocationHandler 400 log - org.apache.flink.hadoop.shaded.com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag., while invoking ApplicationClientProtocolPBClientImpl.getApplicationReport over 17. Trying to failover immediately.
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider 100 performFailover - Failing over to 18
[dlink] 2024-02-02 14:10:46 CST INFO org.apache.hadoop.io.retry.RetryInvocationHandler 400 log - java.net.ConnectException: Call From hadoop80/xxxx to xxxx:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ApplicationClientProtocolPBClientImpl.getApplicationReport over 18 after 1 failover attempts. Trying to failover after sleeping for 25596ms.
[dl
.....
[dlink] 2024-02-02 11:47:16 CST INFO org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider 100 performFailover - Failing over to 17
[dlink] 2024-02-02 11:47:16 CST ERROR com.dlink.utils.LogUtil 46 getError - org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster.
at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:460)
at com.dlink.gateway.yarn.YarnPerJobGateway.submitJobGraph(YarnPerJobGateway.java:100)
at com.dlink.job.JobManager.submitByGateway(JobManager.java:629)
at com.dlink.job.JobManager.executeSql(JobManager.java:486)
at com.dlink.service.impl.TaskServiceImpl.submitTask(TaskServiceImpl.java:251)
at com.dlink.service.impl.TaskServiceImpl$$FastClassBySpringCGLIB$$168bea78.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
at com.dlink.aop.UdfClassLoaderAspect.round(UdfClassLoaderAspect.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
at com.dlink.service.impl.TaskServiceImpl$$EnhancerBySpringCGLIB$$49950b8c.submitTask()
at com.dlink.controller.TaskController.submit(TaskController.java:127)
at com.dlink.controller.TaskController$$FastClassBySpringCGLIB$$a86993b6.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
at com.dlink.controller.TaskController$$EnhancerBySpringCGLIB$$6629eec0.submit()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1071)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:964)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:696)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:124)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.hadoop.shaded.com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
at org.apache.flink.hadoop.shaded.com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
at org.apache.flink.hadoop.shaded.com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
at org.apache.flink.hadoop.shaded.com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:310)
at org.apache.hadoop.yarn.proto.YarnProtos$ApplicationReportProto.(YarnProtos.java:15775)
at org.apache.hadoop.yarn.proto.YarnProtos$ApplicationReportProto.(YarnProtos.java:15551)
at org.apache.hadoop.yarn.proto.YarnProtos$ApplicationReportProto$1.parsePartialFrom(YarnProtos.java:15826)
at org.apache.hadoop.yarn.proto.YarnProtos$ApplicationReportProto$1.parsePartialFrom(YarnProtos.java:15821)
at org.apache.flink.hadoop.shaded.com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
at org.apache.hadoop.yarn.proto.YarnServiceProtos$GetApplicationReportResponseProto.(YarnServiceProtos.java:15821)
at org.apache.hadoop.yarn.proto.YarnServiceProtos$GetApplicationReportResponseProto.(YarnServiceProtos.java:15768)
at org.apache.hadoop.yarn.proto.YarnServiceProtos$GetApplicationReportResponseProto$1.parsePartialFrom(YarnServiceProtos.java:15859)
at org.apache.hadoop.yarn.proto.YarnServiceProtos$GetApplicationReportResponseProto$1.parsePartialFrom(YarnServiceProtos.java:15854)
at org.apache.flink.hadoop.shaded.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:89)
at org.apache.flink.hadoop.shaded.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:95)
at org.apache.flink.hadoop.shaded.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
at org.apache.hadoop.ipc.RpcWritable$ProtobufWrapper.readFrom(RpcWritable.java:125)
at org.apache.hadoop.ipc.RpcWritable$Buffer.getValue(RpcWritable.java:187)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.getReturnMessage(ProtobufRpcEngine.java:283)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:269)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy393.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:228)
at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:411)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:348)
at com.sun.proxy.$Proxy394.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:480)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271)
at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1059)
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:558)
at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:453)
... 92 more

[Zzm0809]

1. 0.7.x is no longer maintained. You can repair it yourself. You can try version 1.0.
2. Community members do not have such an environment and cannot reproduce it. It is recommended to debug and fix it by yourself based on the 0.7 branch debug.

[github-actions]

Hello @rollerQ, thank you for your feedback, but this issue will not be fixed. You can search for keywords in the Issue list to view it. Issues are marked `Wont Fix`

你好 @rollerQ, 感谢你的反馈, 但是此问题不会被修复，你可以在 Issue 列表中搜索关键字来查看, Issues 标注为 `Wont Fix` 。

[aiwenmo]

Thanks for your feedback.
The cause of the problem may be dependency conflicts. Unfortunately, due to the lack of corresponding integrated environments, the community currently has no plans to address dependency issues in third-party environments.

[zackyoungh]

You replace the hdfs:///fink/lib to hdfs:/// flow/lib to try it

[rollerQ]

Thanks for all responses to this question