-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix myrocks_hotbackup for python3 #3629
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the contribution.
Can you rebase this to 10.5.
Also first commit should be titled more like the pr - 'fix myrocksdb_hostbackup for python3`. Can you also mention the python version it is now finally failing with.
Can the commit be squashed up too. Micro changes aren't adding value to the commit history.
storage/rocksdb/myrocks_hotbackup.py
Outdated
exclude_files = ['master.info', 'relay-log.info', 'worker-relay-log.info', | ||
'auto.cnf', 'gaplock.log', 'ibdata', 'ib_logfile', '.trash'] | ||
exclude_files = [b'master.info', b'relay-log.info', b'worker-relay-log.info', | ||
b'auto.cnf', b'gaplock.log', b'ibdata', b'ib_logfile', b'.trash'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you describe the significance of this? It looks like a backwards/forwards utf8 coding/decoding.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In python3, I don't know why, db and filename is bytes , not str.
After I modified it like this, it can run.Otherwise, an error will be reported.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By the way, I have a question.
I use this can recover data , but failed on MariaDB 11.5.2 boot
recover cmd
DATADIR=/mnt/data/i18n/mariadb/data
systemctl stop mariadb
/usr/bin/python3 /usr/local/mysql/bin/myrocks_hotbackup --move_back \
--datadir=$DATADIR --rocksdb_datadir=$DATADIR \
--rocksdb_waldir=$DATADIR --backup_dir=/tmp/myrocks.bak
chown -R mysql:mysql $DATADIR
systemctl start mariadb
failed log
2024-11-13 12:53:00 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table i18n.authUidMail, but that table is not registered in RocksDB
2024-11-13 12:53:00 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table i18n.authUa, but that table is not registered in RocksDB
2024-11-13 12:53:00 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table i18n.authHardware, but that table is not registered in RocksDB
2024-11-13 12:53:00 0 [Warning] RocksDB: Schema mismatch - A .frm file exists for table i18n.uidHost, but that table is not registered in RocksDB
2024-11-13 12:53:00 0 [ERROR] RocksDB: Problems validating data dictionary against .frm files, exiting
2024-11-13 12:53:00 0 [ERROR] RocksDB: Failed to initialize DDL manager.
2024-11-13 12:53:00 0 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.
2024-11-13 12:53:00 0 [Note] Plugin 'InnoDB' is disabled.
2024-11-13 12:53:00 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-11-13 12:53:00 0 [Note] Plugin 'wsrep-provider' is disabled.
2024-11-13 12:53:00 0 [Note] Semi-sync replication enabled on the master.
2024-11-13 12:53:00 0 [ERROR] Unknown/unsupported storage engine: rocksdb
2024-11-13 12:53:00 0 [ERROR] Aborting
ls dir
❯ ls /mnt/data/i18n/mariadb/data
#rocksdb 000265.sst 000269.sst 000272.sst aria_log_control i18n mariadb_upgrade_info OPTIONS-000251 sys
000261.sst 000267.sst 000271.sst aria_log.00000001 CURRENT MANIFEST-000244 mysql performance_schema
/tmp/myrocks.bak
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I originally wanted to use mariadb-backup to back up myrocks, but found that it was impossible to back up myrocks without InnoDB.
For details, see:
https://lists.mariadb.org/hyperkitty/list/[email protected]/thread/7Z64ZTVLUESRSTU5QIED4OLNOCMT5DON/
I tried to use myrocks_hotbackup to backup and restore, but got stuck again. A .frm file exists for table i18n.uidHost, but that table is not registered in RocksDB
this is frustrating
fix /usr/bin/python3 /usr/local/mysql/bin/myrocks_hotbackup --move_back --datadir=$DATADIR --rocksdb_datadir="$DATADIR/#rocksdb" --rocksdb_waldir=$DATADIR --backup_dir=/tmp/myrocks.bak 2024-11-13 14:00:35.415 ERROR Directory /mnt/data/i18n/mariadb/data has file or directory #rocksdb!
see https://peps.python.org/pep-0469/