Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export Parquet Tablet Audio-Set is null bytes in Arrow #6923

Open
anioji opened this issue May 27, 2024 · 0 comments
Open

Export Parquet Tablet Audio-Set is null bytes in Arrow #6923

anioji opened this issue May 27, 2024 · 0 comments

Comments

@anioji
Copy link

anioji commented May 27, 2024

Describe the bug

Exporting the processed audio inside the table with the dataset.to_parquet function, the object pyarrow {bytes: null, path: "Some/Path"}
At the same time, the same dataset uploaded to the hub has bit arrays

Screenshot from 2024-05-27 19-14-49
Screenshot from 2024-05-27 19-12-51

Steps to reproduce the bug

1.Get dataset from audio and cast it
2.Export and push dataset
3.It’s scary to be indignant at the difference in the uploaded dataset and the fact that it was saved locally

from datasets import Dataset, Audio 
df = Dataset.from_csv("./datasets.csv")
df = df.cast_column("audio", Audio(16000))
df.to_parquet("./datasets.parquet")
df.push_to_hub(repo_id="************", token="**********************")

You can use "try replicate case" for this
replicate_packet.zip

Expected behavior

Two parquet tables identical in content. It is obvious?

Environment info

Python 3.11+ (I try did it in 3.12 and got same result )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant