Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support backup and restore parameters #8472

Merged
merged 15 commits into from
Nov 29, 2024

Conversation

gnolong
Copy link
Contributor

@gnolong gnolong commented Nov 15, 2024

fix #8475
Backup and restore tools like pg_dump, pg_restore, mysqldump, etc., require various configuration parameters when performing database-level, schema-level, or table-level tasks, and these parameter structures differ among database engines. Therefore, we add parametersSchema to the actionSet API to define the schema of necessary configuration parameters, and add parameters to the backup and restore API to pass parameter values to backup or restore workloads through environment variables.

  • actionset API
    The actionset API introduces the parametersSchema structure to define the parameter schema for backup and restore tools. The withParameters method specifies the parameters required for backup or restore operations.

  • on-demand backup
    A read-only parameters structure has been added under backup.spec in the backup API to pass configuration values defined by parametersSchema.

  • scheduled backup
    The schedulePolicy structure needs to include a parameters structure. The cron job created by backupSchedule will regularly execute on-demand backups with the corresponding parameters structure. If multiple parameter configurations are needed, new schedulePolicy entries with different parameters structures can be appended to the schedulePolicy array. Additionally, a name field has been added to schedulePolicy to indicate different configurations.

  • opsRequest API
    The opsRequest.spec.restore and opsRequest.spec.backup sections need to include a parameters structure for generating the corresponding restore and backup.

@github-actions github-actions bot added the size/XXL Denotes a PR that changes 1000+ lines. label Nov 15, 2024
Copy link

codecov bot commented Nov 18, 2024

Codecov Report

Attention: Patch coverage is 32.33831% with 136 lines in your changes missing coverage. Please review.

Project coverage is 61.92%. Comparing base (108f7fc) to head (b4c434d).
Report is 13 commits behind head on main.

Files with missing lines Patch % Lines
pkg/dataprotection/utils/utils.go 0.00% 40 Missing ⚠️
pkg/dataprotection/backup/scheduler.go 27.02% 25 Missing and 2 partials ⚠️
pkg/dataprotection/backup/utils.go 0.00% 16 Missing ⚠️
pkg/dataprotection/utils/backup.go 0.00% 13 Missing ⚠️
pkg/dataprotection/restore/utils.go 0.00% 11 Missing ⚠️
controllers/dataprotection/actionset_controller.go 78.12% 4 Missing and 3 partials ⚠️
...trollers/apps/transformer_cluster_backup_policy.go 60.00% 5 Missing and 1 partial ⚠️
pkg/dataprotection/utils/envvar.go 0.00% 6 Missing ⚠️
.../dataprotection/backuppolicytemplate_controller.go 69.23% 2 Missing and 2 partials ⚠️
pkg/controller/plan/restore.go 33.33% 3 Missing and 1 partial ⚠️
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8472      +/-   ##
==========================================
+ Coverage   60.55%   61.92%   +1.36%     
==========================================
  Files         357      376      +19     
  Lines       42577    46626    +4049     
==========================================
+ Hits        25784    28872    +3088     
- Misses      14476    15206     +730     
- Partials     2317     2548     +231     
Flag Coverage Δ
unittests 61.92% <32.33%> (+1.36%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@gnolong gnolong marked this pull request as ready for review November 18, 2024 07:42
@gnolong gnolong requested review from ldming, wangyelei, zjx20 and a team as code owners November 18, 2024 07:42
apis/dataprotection/v1alpha1/backup_types.go Outdated Show resolved Hide resolved
apis/dataprotection/v1alpha1/backupschedule_types.go Outdated Show resolved Hide resolved
controllers/apps/transformer_cluster_backup_policy.go Outdated Show resolved Hide resolved
controllers/dataprotection/backup_controller.go Outdated Show resolved Hide resolved
pkg/dataprotection/utils/backup.go Outdated Show resolved Hide resolved
pkg/dataprotection/utils/backup.go Outdated Show resolved Hide resolved
pkg/dataprotection/restore/utils.go Show resolved Hide resolved
@apecloud-bot apecloud-bot added the approved PR Approved Test label Nov 26, 2024
@apecloud-bot apecloud-bot removed the approved PR Approved Test label Nov 26, 2024
pkg/dataprotection/restore/utils.go Outdated Show resolved Hide resolved
apis/dataprotection/v1alpha1/types.go Outdated Show resolved Hide resolved
controllers/dataprotection/backup_controller_test.go Outdated Show resolved Hide resolved
pkg/testutil/dataprotection/utils.go Outdated Show resolved Hide resolved
controllers/dataprotection/actionset_controller_test.go Outdated Show resolved Hide resolved
controllers/dataprotection/actionset_controller_test.go Outdated Show resolved Hide resolved
pkg/dataprotection/backup/utils.go Outdated Show resolved Hide resolved
pkg/dataprotection/backup/utils.go Outdated Show resolved Hide resolved
pkg/dataprotection/utils/utils.go Outdated Show resolved Hide resolved
@gnolong gnolong requested review from zjx20 and ldming November 28, 2024 08:39
@apecloud-bot apecloud-bot added the approved PR Approved Test label Nov 29, 2024
@gnolong gnolong merged commit a68fb12 into main Nov 29, 2024
37 checks passed
@gnolong gnolong deleted the support/backup-restore-parameters branch November 29, 2024 03:29
@github-actions github-actions bot added this to the Release 0.9.2 milestone Nov 29, 2024
wangyelei pushed a commit that referenced this pull request Nov 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved PR Approved Test area/user-interaction size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Features] support backup and restore parameters to implement database and table-level backups
5 participants