[Bug] [CDCSOURCE] MysqlCDC to paimon with hive catalog error #3722

zhuxt2015 · 2024-08-20T09:05:06Z

Search before asking

I had searched in the issues and found no similar issues.

What happened

SQL

SET 'table.exec.sink.upsert-materialize' = 'NONE'; -- 消除无用的SinkMaterialize算子。

EXECUTE CDCSOURCE dinky_paimon_auto_create_table
WITH
  (
    'connector' = 'mysql-cdc',
    'hostname' = 'mysql-new',
    'port' = '3306',
    'username' = 'root',
    'password' = 'root',
    'checkpoint' = '10000',
    'parallelism' = '1',
    'jdbc.properties.useSSL' = 'false',
    'jdbc.properties.tinyInt1isBit' = 'false',
    'scan.startup.mode' = 'initial',
    'table-name' = 'dinky_dev\.dinky_history',
'sink.connector' = 'sql-catalog',
'sink.catalog.name' = 'paimon_test',
'sink.catalog.type' = 'paimon',
'sink.catalog.metastore' = 'hive',
'sink.catalog.uri' = 'thrift://cdhtestdn02:9083',
'sink.catalog.warehouse'='hdfs:///user/hive/warehouse',
'sink.catalog.hive-conf-dir' = '/etc/hive/conf',
'sink.catalog.hadoop-conf-dir' = '/etc/hadoop/conf'
  );

error log:

Start Process:FlinkSubmit/12
Start Process Step:SUBMIT_PRECHECK
2024-08-20 07:55:33.205 INFO  org.dinky.service.impl.TaskServiceImpl(188): Start check and config task, task:test-3
Process Step SUBMIT_PRECHECK exit with status:FINISHED
Start Process Step:SUBMIT_EXECUTE
Start Process Step:SUBMIT_BUILD_CONFIG
2024-08-20 07:55:33.211 INFO  org.dinky.service.impl.TaskServiceImpl(300): Start initialize FlinkSQLEnv:
2024-08-20 07:55:33.212 INFO  org.dinky.service.impl.TaskServiceImpl(320): Initializing data permissions...
2024-08-20 07:55:33.220 INFO  org.dinky.service.impl.TaskServiceImpl(322): Finish initialize FlinkSQLEnv.
2024-08-20 07:55:33.221 INFO  org.dinky.service.impl.TaskServiceImpl(250): Init remote cluster
Process Step SUBMIT_BUILD_CONFIG exit with status:FINISHED
2024-08-20 07:55:33.263 INFO  org.dinky.service.task.FlinkSqlTask(67): Initializing Flink job config...
2024-08-20 07:55:33.277 INFO  org.dinky.job.builder.JobUDFBuilder(125): A total of 0 UDF have been Init.
2024-08-20 07:55:33.278 INFO  org.dinky.job.builder.JobUDFBuilder(126): Initializing Flink UDF...Finish
2024-08-20 07:55:33.279 INFO  org.dinky.trans.AbstractOperation(77): Start build CDCSOURCE Task...
2024-08-20 07:55:33.303 INFO  org.dinky.trans.AbstractOperation(174): A total of 1 tables were detected...
2024-08-20 07:55:33.304 INFO  org.dinky.trans.AbstractOperation(176): 1: dinky_dev.dinky_history
2024-08-20 07:55:33.304 INFO  org.dinky.trans.AbstractOperation(185): Set parallelism: 1
2024-08-20 07:55:33.305 INFO  org.dinky.trans.AbstractOperation(189): Set checkpoint: 10000
2024-08-20 07:55:33.349 INFO  org.dinky.trans.AbstractOperation(192): Build mysql-cdc successful...
2024-08-20 07:55:33.350 INFO  org.dinky.cdc.AbstractSinkBuilder(90): Build catalog successful...
2024-08-20 07:55:33.497 INFO  org.apache.paimon.hive.HiveCatalog(654): Setting hive conf dir as /etc/hive/conf
2024-08-20 07:55:33.508 INFO  org.apache.hadoop.hive.conf.HiveConf(187): Found configuration file file:/etc/hive/conf/hive-site.xml
2024-08-20 07:55:34.495 INFO  hive.metastore(405): Trying to connect to metastore with URI thrift://cdhtestdn02.yili.com:9083
2024-08-20 07:55:34.583 INFO  hive.metastore(479): Opened a connection to metastore, current connections: 1
2024-08-20 07:55:34.659 INFO  hive.metastore(532): Connected to metastore.
2024-08-20 07:55:35.063 INFO  org.dinky.cdc.AbstractSinkBuilder(220): Build deserialize successful...
2024-08-20 07:55:35.070 INFO  org.dinky.cdc.AbstractSinkBuilder(146): Build dinky_dev.dinky_history shunt successful...
2024-08-20 07:55:35.072 INFO  org.dinky.cdc.AbstractSinkBuilder(153): Build dinky_dev.dinky_history flatMap successful...
2024-08-20 07:55:35.073 INFO  org.dinky.cdc.AbstractSinkBuilder(154): Start build dinky_dev.dinky_history sink...
2024-08-20 07:55:35.075 INFO  org.dinky.cdc.AbstractSinkBuilder(74): Create VIEW_dinky_dev_dinky_history temporaryView successful...
2024-08-20 07:55:35.076 INFO  org.dinky.cdc.AbstractSinkBuilder(248): INSERT INTO `paimon_test.dinky_dev.dinky_history` SELECT
    `id` 
    ,`tenant_id` 
    ,`cluster_id` 
    ,`cluster_configuration_id` 
    ,`session` 
    ,`job_id` 
    ,`job_name` 
    ,`job_manager_address` 
    ,`status` 
    ,`batch_model` 
    ,`type` 
    ,`statement` 
    ,`error` 
    ,`result` 
    ,`config_json` 
    ,`start_time` 
    ,`end_time` 
    ,`task_id` 
 FROM `VIEW_dinky_dev_dinky_history`
2024-08-20 07:55:35.083 ERROR org.dinky.cdc.AbstractSinkBuilder(158): Build dinky_dev.dinky_history cdc sync failed...
2024-08-20 07:55:35.084 ERROR org.dinky.cdc.AbstractSinkBuilder(159): org.apache.flink.table.api.TableException: Cannot find table '`default_catalog`.`default_database`.`paimon_test.dinky_dev.dinky_history`' in any of the catalogs [paimon_test, default_catalog], nor as a temporary table.
	at org.apache.flink.table.catalog.CatalogManager.lambda$getTableOrError$4(CatalogManager.java:622)
	at java.util.Optional.orElseThrow(Optional.java:290)
	at org.apache.flink.table.catalog.CatalogManager.getTableOrError(CatalogManager.java:619)
	at org.apache.flink.table.planner.operations.SqlNodeToOperationConversion.convertSqlInsert(SqlNodeToOperationConversion.java:741)
	at org.apache.flink.table.planner.operations.SqlNodeToOperationConversion.convertValidatedSqlNode(SqlNodeToOperationConversion.java:353)
	at org.apache.flink.table.planner.operations.SqlNodeToOperationConversion.convert(SqlNodeToOperationConversion.java:262)
	at org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:106)
	at org.dinky.executor.ParserWrapper.parse(ParserWrapper.java:47)
	at org.dinky.cdc.AbstractSinkBuilder.createInsertOperations(AbstractSinkBuilder.java:250)
	at org.dinky.cdc.sql.catalog.SQLCatalogSinkBuilder.addTableSink(SQLCatalogSinkBuilder.java:76)
	at org.dinky.cdc.sql.AbstractSqlSinkBuilder.lambda$addTableSinkForTags$0(AbstractSqlSinkBuilder.java:156)
	at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
	at org.dinky.cdc.sql.AbstractSqlSinkBuilder.addTableSinkForTags(AbstractSqlSinkBuilder.java:142)
	at org.dinky.cdc.sql.AbstractSqlSinkBuilder.build(AbstractSqlSinkBuilder.java:244)
	at org.dinky.trans.ddl.CreateCDCSourceOperation.execute(CreateCDCSourceOperation.java:193)
	at org.dinky.interceptor.FlinkInterceptor.build(FlinkInterceptor.java:53)
	at org.dinky.executor.Executor.pretreatExecute(Executor.java:186)
	at org.dinky.executor.Executor.executeSql(Executor.java:199)
	at org.dinky.job.builder.JobExecuteBuilder.run(JobExecuteBuilder.java:96)
	at org.dinky.job.JobManager.executeSql(JobManager.java:301)
	at org.dinky.service.task.FlinkSqlTask.execute(FlinkSqlTask.java:68)
	at org.dinky.service.impl.TaskServiceImpl.executeJob(TaskServiceImpl.java:202)
	at org.dinky.service.impl.TaskServiceImpl$$FastClassBySpringCGLIB$$22087f7c.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
	at org.dinky.aop.ProcessAspect.processStepAround(ProcessAspect.java:110)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707)
	at org.dinky.service.impl.TaskServiceImpl$$EnhancerBySpringCGLIB$$5d452f2f.executeJob(<generated>)
	at org.dinky.service.impl.TaskServiceImpl.submitTask(TaskServiceImpl.java:333)
	at org.dinky.service.impl.TaskServiceImpl$$FastClassBySpringCGLIB$$22087f7c.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
	at org.dinky.aop.UdfClassLoaderAspect.round(UdfClassLoaderAspect.java:55)
	at sun.reflect.GeneratedMethodAccessor284.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707)
	at org.dinky.service.impl.TaskServiceImpl$$EnhancerBySpringCGLIB$$5d452f2f.submitTask(<generated>)
	at org.dinky.controller.TaskController.submitTask(TaskController.java:88)
	at org.dinky.controller.TaskController$$FastClassBySpringCGLIB$$5b68bd32.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
	at org.dinky.aop.TaskOperationPermissionAspect.processAround(TaskOperationPermissionAspect.java:82)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89)
	at org.dinky.aop.ProcessAspect.processAround(ProcessAspect.java:73)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707)
	at org.dinky.controller.TaskController$$EnhancerBySpringCGLIB$$71bcfe79.submitTask(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:529)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:114)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at cn.dev33.satoken.filter.SaPathCheckFilterForServlet.doFilter(SaPathCheckFilterForServlet.java:55)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:765)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:750)

2024-08-20 07:55:35.086 INFO  org.dinky.cdc.AbstractSinkBuilder(250): A total of 0 table cdc sync were build successful...
2024-08-20 07:55:35.086 INFO  org.dinky.trans.AbstractOperation(195): Build CDCSOURCE Task successful!
2024-08-20 07:55:35.159 INFO  org.apache.flink.client.program.rest.RestClusterClient(410): Submitting job 'test-3' (97833154b6a4650869f688a478c929b4).
2024-08-20 07:55:35.249 INFO  org.apache.flink.client.program.rest.RestClusterClient(429): Successfully submitted job 'test-3' (97833154b6a4650869f688a478c929b4) to 'http://flink-session-rest:8081'.
2024-08-20 07:55:35.470 INFO  org.dinky.service.impl.TaskServiceImpl(203): execute job finished,status is SUCCESS
Process Step SUBMIT_EXECUTE exit with status:FINISHED
2024-08-20 07:55:35.471 INFO  org.dinky.service.impl.TaskServiceImpl(337): Job Submit success
Process FlinkSubmit/12 exit with status:FINISHED

What you expected to happen

create paimon table with hive catalog automate, can query paimon table in hive CLI

How to reproduce

use example SQL

Anything else

No response

Version

dev

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

github-actions · 2024-08-20T09:05:23Z

Hello @zhuxt2015, this issue is about web, so I assign it to @Zzm0809. If you have any questions, you can comment and reply.

你好 @zhuxt2015, 这个 issue 是关于 web 的，所以我把它分配给了 @Zzm0809。如有任何问题，可以评论回复。

github-actions · 2024-08-20T09:05:24Z

Hello @zhuxt2015, this issue is about CDC/CDCSOURCE, so I assign it to @aiwenmo. If you have any questions, you can comment and reply.

你好 @zhuxt2015, 这个 issue 是关于 CDC/CDCSOURCE 的，所以我把它分配给了 @aiwenmo。如有任何问题，可以评论回复。

Zzm0809 · 2024-08-20T12:03:33Z

看一下你的报错信息:
2024-08-20 07:55:35.084 ERROR org.dinky.cdc.AbstractSinkBuilder(159): org.apache.flink.table.api.TableException: Cannot find table 'default_catalog.default_database.paimon_test.dinky_dev.dinky_history' in any of the catalogs [paimon_test, default_catalog], nor as a temporary table.

你的表层级: default_catalog.default_database.paimon_test.dinky_dev.dinky_history 这肯定是不对的

zhuxt2015 · 2024-08-21T01:05:10Z

看一下你的报错信息:

2024-08-20 07:55:35.084 ERROR org.dinky.cdc.AbstractSinkBuilder(159): org.apache.flink.table.api.TableException: Cannot find table 'default_catalog.default_database.paimon_test.dinky_dev.dinky_history' in any of the catalogs [paimon_test, default_catalog], nor as a temporary table.

你的表层级: default_catalog.default_database.paimon_test.dinky_dev.dinky_history 这肯定是不对的

这个表表层级是dinky生成的，我想基于hive catalog创建一个default.db/dinky_history paimon表，应该怎么做呢？

diaozhonghao · 2024-09-05T04:53:00Z

看一下你的报错信息:
2024-08-20 07:55:35.084 ERROR org.dinky.cdc.AbstractSinkBuilder(159): org.apache.flink.table.api.TableException: Cannot find table 'default_catalog.default_database.paimon_test.dinky_dev.dinky_history' in any of the catalogs [paimon_test, default_catalog], nor as a temporary table.
你的表层级: default_catalog.default_database.paimon_test.dinky_dev.dinky_history 这肯定是不对的

这个表表层级是dinky生成的，我想基于hive catalog创建一个default.db/dinky_history paimon表，应该怎么做呢？

解决了吗哥们,我也sink到hive也遇到了同样的问题

zhuxt2015 · 2024-09-05T05:06:38Z

看一下你的报错信息:

2024-08-20 07:55:35.084 ERROR org.dinky.cdc.AbstractSinkBuilder(159): org.apache.flink.table.api.TableException: Cannot find table 'default_catalog.default_database.paimon_test.dinky_dev.dinky_history' in any of the catalogs [paimon_test, default_catalog], nor as a temporary table.

你的表层级: default_catalog.default_database.paimon_test.dinky_dev.dinky_history 这肯定是不对的

这个表表层级是dinky生成的，我想基于hive catalog创建一个default.db/dinky_history paimon表，应该怎么做呢？

解决了吗哥们,我也sink到hive也遇到了同样的问题

我现在不用整库同步了，改用单表同步

diaozhonghao · 2024-09-05T14:09:32Z

看一下你的报错信息:

2024-08-20 07:55:35.084 ERROR org.dinky.cdc.AbstractSinkBuilder(159): org.apache.flink.table.api.TableException: Cannot find table 'default_catalog.default_database.paimon_test.dinky_dev.dinky_history' in any of the catalogs [paimon_test, default_catalog], nor as a temporary table.

你的表层级: default_catalog.default_database.paimon_test.dinky_dev.dinky_history 这肯定是不对的

这个表表层级是dinky生成的，我想基于hive catalog创建一个default.db/dinky_history paimon表，应该怎么做呢？

解决了吗哥们,我也sink到hive也遇到了同样的问题

我现在不用整库同步了，改用单表同步

okok 等我研究出来了我给你反馈奥哥们

github-actions · 2024-11-01T00:02:12Z

Hello @, this issue has not been active for more than 30 days. This issue will be closed in 7 days if there is no response. If you have any questions, you can comment and reply.

你好 @, 这个 issue 30 天内没有活跃，7 天后将关闭，如需回复，可以评论回复。

Zzm0809 · 2024-11-07T10:46:08Z

已修复

zhuxt2015 added Bug Something isn't working Waiting for reply Waiting for reply labels Aug 20, 2024

github-actions bot assigned Zzm0809 Aug 20, 2024

github-actions bot assigned aiwenmo Aug 20, 2024

Zzm0809 removed the Waiting for reply Waiting for reply label Sep 7, 2024

github-actions bot added the Invalid Invalid label Nov 1, 2024

Zzm0809 closed this as completed Nov 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] [CDCSOURCE] MysqlCDC to paimon with hive catalog error #3722

[Bug] [CDCSOURCE] MysqlCDC to paimon with hive catalog error #3722

zhuxt2015 commented Aug 20, 2024

github-actions bot commented Aug 20, 2024

github-actions bot commented Aug 20, 2024

Zzm0809 commented Aug 20, 2024 •

edited

Loading

zhuxt2015 commented Aug 21, 2024

diaozhonghao commented Sep 5, 2024

zhuxt2015 commented Sep 5, 2024

diaozhonghao commented Sep 5, 2024

github-actions bot commented Nov 1, 2024

Zzm0809 commented Nov 7, 2024

[Bug] [CDCSOURCE] MysqlCDC to paimon with hive catalog error #3722

[Bug] [CDCSOURCE] MysqlCDC to paimon with hive catalog error #3722

Comments

zhuxt2015 commented Aug 20, 2024

Search before asking

What happened

What you expected to happen

How to reproduce

Anything else

Version

Are you willing to submit PR?

Code of Conduct

github-actions bot commented Aug 20, 2024

github-actions bot commented Aug 20, 2024

Zzm0809 commented Aug 20, 2024 • edited Loading

zhuxt2015 commented Aug 21, 2024

diaozhonghao commented Sep 5, 2024

zhuxt2015 commented Sep 5, 2024

diaozhonghao commented Sep 5, 2024

github-actions bot commented Nov 1, 2024

Zzm0809 commented Nov 7, 2024

Zzm0809 commented Aug 20, 2024 •

edited

Loading