Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backport schema feature from dolphinbeat #358

Open
wants to merge 19 commits into
base: master
Choose a base branch
from

Conversation

bytewatch
Copy link
Contributor

Backporting schema feature from dolphinbeat.

Feature:

  • Save schema info by calling mysqldump when first start. Save into boltdb default.
  • Replay DDL statement when meet DDL in binlog.

Warning:

  • Don‘t soupport backword compatibility, you need to do something manually to upgrage, see below.

TODO:

  • Provide a tool to make upgrade from previous version of go-mysql/canal, by saving schema info explicitly .
  • Provide a tool to handle problems when replay DDL failed, like dolphinbeat's ddl/exec admin API

canal/canal.go Outdated
@@ -11,13 +11,13 @@ import (
"sync"
"time"

"github.com/bytewatch/dolphinbeat/schema"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Em, can you put all to go-mysql and not import any dolphinbeat? It is very strange for circular dependence. E.g, go-mysql depends dolphinebeat, and dolphinebeat depends on go-mysql.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have copied the source from dolpinbeat/schema to go-mysql/schema. Now I have a build problem under go 1.9 becuase pingcap/parser depends on strings.Builder: Build 655. Do you have any idea?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bytewatch

I think we don't need to consider go 1.9 now, we will support at 1.11 at least later.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will support Go mod and remove vendor at first.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's ok to build now, PTAL. By the way, can i ask a question about TiDB's Percolator implemention? I wonder can it turn to 1PC if all mutations is in one region to improve perfermance?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bytewatch

There is a corner-case for 1 PC, and if we want to do this, we still need to do some extra works, so this has a lower priority for us. If you want to have a try, you can discuss with us in TiKV issues directly.

@@ -61,8 +65,6 @@ func (s *canalTestSuite) SetUpSuite(c *C) {
s.execute(c, "DELETE FROM test.canal_test")
s.execute(c, "INSERT INTO test.canal_test (content, name) VALUES (?, ?), (?, ?), (?, ?)", "1", "a", `\0\ndsfasdf`, "b", "", "c")

s.execute(c, "SET GLOBAL binlog_format = 'ROW'")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why remove this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because my account dosen't have super privilege to set this global variable. And I think it's ok if mysql is configured to use ROW format already, like the travis config.

@domyway
Copy link
Contributor

domyway commented May 28, 2019

还没有合并进去么,非常期待这个优化 @siddontang @bytewatch

@IANTHEREAL
Copy link
Collaborator

IANTHEREAL commented Aug 30, 2019

@bytewatch hi, you haven’t updated this pr for a long time, and I just want to know what is your plan now?

@IANTHEREAL
Copy link
Collaborator

@domyway I think I can push it.

@IANTHEREAL
Copy link
Collaborator

IANTHEREAL commented Aug 31, 2019

@3pointer now please take charge of this pr

@dveeden
Copy link
Collaborator

dveeden commented Nov 8, 2024

@bytewatch is this still something you are interested in? If so can we resolve the conflicts and/or rebase on the latest master?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants