Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Region is not balanced when Lighting import million tables #58508

Open
lilinghai opened this issue Dec 24, 2024 · 0 comments
Open

Region is not balanced when Lighting import million tables #58508

lilinghai opened this issue Dec 24, 2024 · 0 comments
Labels
type/bug The issue is confirmed as a bug.

Comments

@lilinghai
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

6 tikv
Lighting import 1 million tables (1w rows each table)
Image

### tidb-lightning configuration
[lightning]
# check if the cluster satisfies the minimum requirement before starting
check-requirements = false
index-concurrency = 16
table-concurrency = 16

[mydumper]
no-schema = true
case-sensitive = false
data-source-dir = "xxx"

# https://docs.pingcap.com/zh/tidb/stable/tidb-lightning-configuration
[conflict]
strategy = ""

[tikv-importer]
backend = "local"
sorted-kv-dir = "/data/sorted-kv-dir"
# parallel-import = true
disk-quota = "3TB"
range-concurrency = 4

# CSV files are imported according to MySQL LOAD DATA INFILE rules.
[mydumper.csv]
# separator between fields, should be an ASCII character.
separator = ","
# whether the CSV files contain a header. If true, the first line will be skipped
header = true
# whether the CSV contains any NULL value. If true, all columns from CSV cannot be NULL.
not-null = false
# if non-null = false (i.e. CSV can contain NULL), fields equal to this value will be treated as NULL
null = "NULL"
# whether to interpret backslash-escape inside strings.
backslash-escape = false
# if a line ends with a separator, remove it.
trim-last-separator = false

[checkpoint]
enable = true

# post-restore provide some options which will be executed after all kv data has been imported into the tikv cluster.
# the execution order are(if set true): checksum -> analyze
[post-restore]
# if set true, checksum will do ADMIN CHECKSUM TABLE <table> for each table.
# checksum = false
# if set true, analyze will do ANALYZE TABLE <table> for each table.
analyze = false

[tidb]
checksum-table-concurrency = 16

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

Release Version: v8.5.0
Git Commit Hash: cf1bb56
Git Branch: heads/refs/tags/v8.5.0
Go Version: go1.23.4
UTC Build Time: 2024-12-12 16:11:49
Race Enabled: false

@lilinghai lilinghai added the type/bug The issue is confirmed as a bug. label Dec 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug The issue is confirmed as a bug.
Projects
None yet
Development

No branches or pull requests

1 participant