[BugFix][FlinkSQL] V1.1.0 version Keberos bug related repair, SQL SET value does not take effect, etc #3875

chenhaipeng · 2024-10-17T11:04:57Z

Fix the following issues:

Feature:

Added getJobInstanceListapi in APIController.
table.topic.map, using topic:tableName,eg: topic1:tableA;topic2:tableB,tableC, can reduce the creation of topics.

Fix:

FlinkAPI Kerberos might start with http or https.
In ExecutorConfig, if the address starts with http://xx, the splitting is incorrect.
Fixed value passing issue in KafkaSinkBuilder, where DeliveryGuarantee was always set to EXACTLY_ONCE because the default submission is in Dinky.
Fixed the issue where the environment variable set 'xxxx' in MysqlCDCBuilder SQL did not take effect.
Fixed the redirection issue in JobOperator.tsx.

Pending:

#3874 (comment)
PS: Thanks to the community for contributing such a useful tool!

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

1.APIController 新增getJobInstanceListapi 2.table.topic.map , 以topic:tableName ,可以减少topic 创建 fix: 1. FlinkAPI kerberos 可能以http或者https 开头 2. ExecutorConfig , address 如果是http://xx分割不对 3. KafkaSinkBuilder 传值问题 4. JobOperator.tsx 跳转问题

Zzm0809 · 2024-10-18T10:09:50Z

dinky-cdc/dinky-cdc-core/src/main/java/org/dinky/cdc/AbstractSinkBuilder.java

@@ -587,4 +590,26 @@ protected List<String> getPKList(Table table) {
    protected ZoneId getSinkTimeZone() {
        return this.sinkTimeZone;
    }
+
+    protected Map<String, String> getTableTopicMap() {


不应该在 dinky-cdc/dinky-cdc-core/src/main/java/org/dinky/cdc/AbstractSinkBuilder.java 中写此方法因为不属于公共抽象方法建议下沉到 kafkasinkbuilder

有好几个KakfaSinkBuilder, KafkaSinkJsonBuilder ，不放这里需要写好几次

Add AbstractKafkaSinkBuilder.

Add AbstractKafkaSinkBuilder.

it just a single method , Is it necessary to add an AbstractKafkaSinkBuilder class?

If sinks other than kafka use this method, your current code logic is feasible.

aiwenmo

Thanks for your pull request.

aiwenmo · 2024-10-18T15:59:42Z

dinky-cdc/dinky-cdc-core/src/main/java/org/dinky/cdc/AbstractSinkBuilder.java

+            } else {
+                properties.setProperty(entry.getKey(), entry.getValue());


Why set other keys that do not start with an attribute?

because there's no place in the constructor of kafkaSinkBuilder to set properties.

Thx. I get it.

This reverts commit 2f1b13a.

aiwenmo · 2024-10-23T03:39:03Z

#3874 will be fixed by #3889

Zzm0809 · 2024-11-09T15:05:17Z

@chenhaipeng 你好, 请合并主仓库最新代码解决 CI 过程出现的问题, 并将您的 Pr title 进行简单规范, 再次感谢您的贡献

chenhaipeng · 2024-11-11T01:15:25Z

@chenhaipeng 你好, 请合并主仓库最新代码解决 CI 过程出现的问题, 并将您的 Pr title 进行简单规范, 再次感谢您的贡献

OK

aiwenmo

LGTM

…ble.topic.map, using topic:tableName,eg: topic1:tableA;topic2:tableB,tableC, can reduce the creation of topics. [Fix] 1.FlinkAPI Kerberos might start with http or https. 2.In ExecutorConfig, if the address starts with http://xx, the splitting is incorrect. Fixed value passing issue in KafkaSinkBuilder, whereDeliveryGuarantee was always set to EXACTLY_ONCE because the default submission is in Dinky. 3.Fixed the issue where the environment variable set 'xxxx' in MysqlCDCBuilder SQL did not take effect. 4.Fixed the redirection issue in JobOperator.tsx.

Signed-off-by: Zzm0809 <[email protected]>

Co-authored-by: zhuxt2015 <[email protected]>

… used by the JVM via -XX:MaxRAMPercentage (DataLinkDC#3840)

…inkDC#3877)

Co-authored-by: gaoyan1998 <[email protected]>

Signed-off-by: Zzm0809 <[email protected]> Co-authored-by: Zzm0809 <[email protected]>

…ress address DataLinkDC#3871 (DataLinkDC#3887) Co-authored-by: jianjun.xu <[email protected]>

Co-authored-by: gaoyan1998 <[email protected]>

…nkDC#3891)

DataLinkDC#3895)

…e the user experience (DataLinkDC#3854) Signed-off-by: Zzm0809 <[email protected]> Co-authored-by: zackyoungh <[email protected]> Co-authored-by: Zzm0809 <[email protected]> Co-authored-by: gaoyan <[email protected]> Co-authored-by: gaoyan1998 <[email protected]> Co-authored-by: zhangyuhang <[email protected]> Co-authored-by: LUOSHANGJIE\71826 <[email protected]> Co-authored-by: 18216499322 <[email protected]> Co-authored-by: Zzm0809 <[email protected]>

Signed-off-by: Zzm0809 <[email protected]> Co-authored-by: Zzm0809 <[email protected]>

… preview (DataLinkDC#3897)

…kDC#3907)

Co-authored-by: gaoyan1998 <[email protected]>

chenhaipeng · 2024-11-12T09:10:52Z

@chenhaipeng 你好, 请合并主仓库最新代码解决 CI 过程出现的问题, 并将您的 Pr title 进行简单规范, 再次感谢您的贡献

我已经重新合并了，pr comit title 简单修改了

chenhaipeng · 2024-11-13T02:24:41Z

dinky-core/src/main/java/org/dinky/job/builder/JobExecuteBuilder.java

@@ -65,6 +65,8 @@ public void run() throws Exception {
                }
                GatewayResult gatewayResult = null;
                config.addGatewayConfig(executor.getSetConfig());
+                config.addGatewayConfig(
+                        executor.getCustomTableEnvironment().getConfig().getConfiguration());


flink sql set 参数不生效，我加了这一行放到这里是否合理，麻烦确认下

github-actions bot assigned chenhaipeng Oct 17, 2024

github-actions bot requested review from aiwenmo, gaoyan1998, zackyoungh and Zzm0809 October 17, 2024 11:05

github-actions bot changed the title ~~v1.10 版本keberos bug 修复，~~ v1.10 version keberos bug fixed, Oct 17, 2024

chenhaipeng changed the title ~~v1.10 version keberos bug fixed,~~ v1.10 版本keberos bug 相关修复 Oct 17, 2024

chenhaipeng mentioned this pull request Oct 17, 2024

cdc task setting jvm resource parameters does not take effect #3868

Open

3 tasks

chenhaipeng changed the title ~~v1.10 版本keberos bug 相关修复~~ v1.10 版本keberos bug 相关修复, sql set 值不生效等问题 Oct 17, 2024

chenhaipeng mentioned this pull request Oct 17, 2024

[Bug] [zookeeper Kebers] java.security.PrivilegedActionException: javax.security.sasl.SaslException: #3835

Open

3 tasks

chenhaipeng changed the title ~~v1.10 版本keberos bug 相关修复, sql set 值不生效等问题~~ v1.1.0 版本keberos bug 相关修复, sql set 值不生效等问题 Oct 17, 2024

chenhaipeng changed the title ~~v1.1.0 版本keberos bug 相关修复, sql set 值不生效等问题~~ V1.1.0 version Keberos bug related repair, SQL SET value does not take effect, etc. #3875 Oct 17, 2024

Zzm0809 reviewed Oct 18, 2024

View reviewed changes

aiwenmo reviewed Oct 18, 2024

View reviewed changes

donotcoffee added 2 commits October 21, 2024 10:35

fix: remove unnecessary properties

2f1b13a

Revert "fix: remove unnecessary properties"

9fb3ccd

This reverts commit 2f1b13a.

aiwenmo approved these changes Nov 11, 2024

View reviewed changes

donotcoffee and others added 9 commits November 12, 2024 16:27

Merge remote-tracking branch 'upstream/dev' into dev

1bc8fe1

[Improvement] change token key name (DataLinkDC#3865)

6a87846

Signed-off-by: Zzm0809 <[email protected]>

[BugFix]fix job id is null exception in query model (DataLinkDC#3869)

c337fcb

Co-authored-by: zhuxt2015 <[email protected]>

[Improvement][Docker]Limit the maximum percentage of container memory…

e9a1a21

… used by the JVM via -XX:MaxRAMPercentage (DataLinkDC#3840)

[Fix-3876] [cdc] Remove quotation marks when building FlinkSQL (DataL…

99ddc5b

…inkDC#3877)

upgrade cdc to 3.2.0 (DataLinkDC#3878)

1b6d647

Co-authored-by: gaoyan1998 <[email protected]>

[BugFix][DevOps]do not save job instance in query mode (DataLinkDC#3880)

adca57a

Signed-off-by: Zzm0809 <[email protected]> Co-authored-by: Zzm0809 <[email protected]>

[Doc] Add ws doc (DataLinkDC#3882)

a8db496

gaoyan1998 and others added 12 commits November 12, 2024 16:47

[Bug][Web] Fix ws bug (DataLinkDC#3881)

07e3a00

[Feature][dinky-geteway] Feature Obtain job information using the ing…

33c2245

…ress address DataLinkDC#3871 (DataLinkDC#3887) Co-authored-by: jianjun.xu <[email protected]>

[Bug]Fix web package (DataLinkDC#3885)

4519505

add package-lock.json (DataLinkDC#3890)

95a1a17

Co-authored-by: gaoyan1998 <[email protected]>

[hotfix][build] Fix Dinky backend CI workflow with Flink 1.20 (DataLi…

add6705

…nkDC#3891)

[Bug] [dinky-admin] Fix the issue of primary key generation strategy … (

81147fa

DataLinkDC#3895)

[Refactor] Refactor get version function (DataLinkDC#3898)

34c8611

Signed-off-by: Zzm0809 <[email protected]> Co-authored-by: Zzm0809 <[email protected]>

[Feature-3893][core][admin][web] Flink SQL task Support insert result…

960579a

… preview (DataLinkDC#3897)

[Optimization-3906][web] Optimize debug task to preview data (DataLin…

2e6f1f0

…kDC#3907)

[Web]Optimized the new UI (DataLinkDC#3904)

3c77f29

Co-authored-by: gaoyan1998 <[email protected]>

Merge remote-tracking branch 'origin/dev' into dev

e7ed1c5

zackyoungh changed the title ~~V1.1.0 version Keberos bug related repair, SQL SET value does not take effect, etc. #3875~~ [BugFix][FlinkSQL] V1.1.0 version Keberos bug related repair, SQL SET value does not take effect, etc Nov 12, 2024

zackyoungh linked an issue Nov 12, 2024 that may be closed by this pull request

[Bug] [HIVE] Did not correctly interpret the Hive table creation statement, but the older version 0.7.1 works. #3874

Closed

3 tasks

zackyoungh added the Bug Something isn't working label Nov 12, 2024

zackyoungh added this to the 1.2.0 milestone Nov 12, 2024

zackyoungh merged commit 237b45a into DataLinkDC:dev Nov 12, 2024
17 checks passed

chenhaipeng commented Nov 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix][FlinkSQL] V1.1.0 version Keberos bug related repair, SQL SET value does not take effect, etc #3875

[BugFix][FlinkSQL] V1.1.0 version Keberos bug related repair, SQL SET value does not take effect, etc #3875

chenhaipeng commented Oct 17, 2024 •

edited

Loading

Zzm0809 Oct 18, 2024

chenhaipeng Oct 21, 2024

aiwenmo Oct 21, 2024

chenhaipeng Oct 22, 2024

aiwenmo Oct 22, 2024

aiwenmo left a comment

aiwenmo Oct 18, 2024

chenhaipeng Oct 21, 2024

aiwenmo Oct 22, 2024

aiwenmo commented Oct 23, 2024

Zzm0809 commented Nov 9, 2024

chenhaipeng commented Nov 11, 2024

aiwenmo left a comment

chenhaipeng commented Nov 12, 2024

chenhaipeng Nov 13, 2024

		} else {
		properties.setProperty(entry.getKey(), entry.getValue());

[BugFix][FlinkSQL] V1.1.0 version Keberos bug related repair, SQL SET value does not take effect, etc #3875

[BugFix][FlinkSQL] V1.1.0 version Keberos bug related repair, SQL SET value does not take effect, etc #3875

Conversation

chenhaipeng commented Oct 17, 2024 • edited Loading

Brief change log

Verify this pull request

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aiwenmo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aiwenmo commented Oct 23, 2024

Zzm0809 commented Nov 9, 2024

chenhaipeng commented Nov 11, 2024

aiwenmo left a comment

Choose a reason for hiding this comment

chenhaipeng commented Nov 12, 2024

Choose a reason for hiding this comment

chenhaipeng commented Oct 17, 2024 •

edited

Loading