Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pickle_tables argument to load functions #444

Merged
merged 9 commits into from
Aug 14, 2024
Merged

Add pickle_tables argument to load functions #444

merged 9 commits into from
Aug 14, 2024

Conversation

hagenw
Copy link
Member

@hagenw hagenw commented Jul 26, 2024

See audeering/audformat#440 and audeering/audbcards#100

Adds pickle_tables argument to audb.load(), audb.load_to() and audb.load_table() with default value of True. If pickle_tables is True, it will store a table in its original format (csv or parquet) in the cache or in root (in the case of audb.load_to()), but also load the table ones and store as pickle file. This will speed up loading the table a second time, and was the default behavior before.

With pickle_tables is False we provide the option to load tables to the cache, that do not fit into memory, and provide the user the option to avoid loading the whole table when only a few lines should be read.

image

image

image


image

image

@hagenw hagenw changed the title Add pickle_cache argument to load() + load_table() Add pickle_cache argument to load()+load_table() Jul 26, 2024
@hagenw hagenw marked this pull request as draft July 26, 2024 06:51
@hagenw hagenw changed the title Add pickle_cache argument to load()+load_table() Add pickle_tables argument to load()+load_table() Jul 26, 2024
@hagenw hagenw changed the title Add pickle_tables argument to load()+load_table() Add pickle_table argument to load()+load_table() Jul 26, 2024
@hagenw hagenw changed the title Add pickle_table argument to load()+load_table() Add pickle_tables argument to load functions Jul 29, 2024
@hagenw hagenw marked this pull request as ready for review July 29, 2024 08:13
audb/core/load.py Outdated Show resolved Hide resolved
Copy link
Member

@ChristianGeng ChristianGeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is quite clear what happened in this merge request.
Minor wording and documentation bumps have been adressed.

Approving.

@hagenw hagenw merged commit 593776c into main Aug 14, 2024
8 checks passed
@hagenw hagenw deleted the skip-pickle branch August 14, 2024 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants