Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix myrocks_hotbackup for python3 #3629

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Conversation

i18nsite
Copy link

@CLAassistant
Copy link

CLAassistant commented Nov 13, 2024

CLA assistant check
All committers have signed the CLA.

@i18nsite i18nsite changed the title python3 collections.OrderedDict removed iteritems fix myrocks_hotbackup for python3 Nov 13, 2024
Copy link
Member

@grooverdan grooverdan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution.

Can you rebase this to 10.5.

Also first commit should be titled more like the pr - 'fix myrocksdb_hostbackup for python3`. Can you also mention the python version it is now finally failing with.

Can the commit be squashed up too. Micro changes aren't adding value to the commit history.

storage/rocksdb/myrocks_hotbackup.py Outdated Show resolved Hide resolved
storage/rocksdb/myrocks_hotbackup.py Show resolved Hide resolved
exclude_files = ['master.info', 'relay-log.info', 'worker-relay-log.info',
'auto.cnf', 'gaplock.log', 'ibdata', 'ib_logfile', '.trash']
exclude_files = [b'master.info', b'relay-log.info', b'worker-relay-log.info',
b'auto.cnf', b'gaplock.log', b'ibdata', b'ib_logfile', b'.trash']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you describe the significance of this? It looks like a backwards/forwards utf8 coding/decoding.

Copy link
Author

@i18nsite i18nsite Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In python3, I don't know why, db and filename is bytes , not str.

After I modified it like this, it can run.Otherwise, an error will be reported.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Author

@i18nsite i18nsite Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, I have a question.

I use this can recover data , but failed on MariaDB 11.5.2 boot

recover cmd

DATADIR=/mnt/data/i18n/mariadb/data
systemctl stop mariadb
/usr/bin/python3 /usr/local/mysql/bin/myrocks_hotbackup --move_back \
--datadir=$DATADIR --rocksdb_datadir=$DATADIR \
--rocksdb_waldir=$DATADIR --backup_dir=/tmp/myrocks.bak
chown -R mysql:mysql $DATADIR
systemctl start mariadb

failed log

2024-11-13 12:53:00 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table i18n.authUidMail, but that table is not registered in RocksDB
2024-11-13 12:53:00 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table i18n.authUa, but that table is not registered in RocksDB
2024-11-13 12:53:00 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table i18n.authHardware, but that table is not registered in RocksDB
2024-11-13 12:53:00 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table i18n.uidHost, but that table is not registered in RocksDB
2024-11-13 12:53:00 0 [ERROR] RocksDB: Problems validating data dictionary against .frm files, exiting
2024-11-13 12:53:00 0 [ERROR] RocksDB: Failed to initialize DDL manager.
2024-11-13 12:53:00 0 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.
2024-11-13 12:53:00 0 [Note] Plugin 'InnoDB' is disabled.
2024-11-13 12:53:00 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-11-13 12:53:00 0 [Note] Plugin 'wsrep-provider' is disabled.
2024-11-13 12:53:00 0 [Note] Semi-sync replication enabled on the master.
2024-11-13 12:53:00 0 [ERROR] Unknown/unsupported storage engine: rocksdb
2024-11-13 12:53:00 0 [ERROR] Aborting

ls dir

❯ ls /mnt/data/i18n/mariadb/data
#rocksdb    000265.sst  000269.sst  000272.sst         aria_log_control  i18n             mariadb_upgrade_info  OPTIONS-000251      sys
000261.sst  000267.sst  000271.sst  aria_log.00000001  CURRENT           MANIFEST-000244  mysql                 performance_schema
/tmp/myrocks.bak                                                                                                                         

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally wanted to use mariadb-backup to back up myrocks, but found that it was impossible to back up myrocks without InnoDB.

For details, see:
https://lists.mariadb.org/hyperkitty/list/[email protected]/thread/7Z64ZTVLUESRSTU5QIED4OLNOCMT5DON/

I tried to use myrocks_hotbackup to backup and restore, but got stuck again. A .frm file exists for table i18n.uidHost, but that table is not registered in RocksDB

this is frustrating

fix
/usr/bin/python3 /usr/local/mysql/bin/myrocks_hotbackup --move_back --datadir=$DATADIR --rocksdb_datadir="$DATADIR/#rocksdb" --rocksdb_waldir=$DATADIR --backup_dir=/tmp/myrocks.bak
2024-11-13 14:00:35.415 ERROR Directory /mnt/data/i18n/mariadb/data has file or directory #rocksdb!
@i18nsite
Copy link
Author

CleanShot 2024-11-13 at 15 10 28
fixed , I use charset='utf8' for mysql connection , fix is return bytes in python3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants