Remote S3 (MinIO) IAM policies for dvc

[FeryET]

Hello.

I have a distributed MinIO setup using this guide, and I have synced my DVC bucket using rclone sync from my AWS s3 to a bucket in this object storage.

The problem is that currently I cannot pull anything from this self hosted object storage, even though I have the correct credentials and my user has the correct access policy.

The result of dvc pull is this:

ERROR: unexpected error - There were headers present in the request which were not signed: An error occurred (AccessDenied) when calling the GetObject operation: There were headers present in the request which were not signed

My .dvc/config is this:

[core]
    remote = s3storage
['remote "s3storage"']
    url = "s3://bucket"
    endpointurl = https://api.minio.example.com
    access_key_id = [acsess_key]
    secret_access_key = [secret_access_key]

But I cannot pull or push anything to it. I'm utterly confused and there is not a lot of documentation about what DVC needs, and how to set ip up.

[rlamy]

There is some documentation here and on StackOverflow, but I don't know if that helps with your issue.

What's the output of dvc pull -vv and dvc doctor?

[dberenbaum]

Thanks @FeryET . I'm asking if you can try to do aws s3 cp s3://new_remote/... local_path to see if you can manually download the data that you want dvc to pull. This could help confirm whether the issue is with dvc or aws/minio.

Edit: If you can download the data with rclone, that might also help, but if it's not too much trouble to use the aws cli, it's probably the easiest way to ensure the basic aws configuration is correct.

[FeryET]

Thanks @FeryET . I'm asking if you can try to do aws s3 cp s3://new_remote/... local_path to see if you can manually download the data that you want dvc to pull. This could help confirm whether the issue is with dvc or aws/minio.

Edit: If you can download the data with rclone, that might also help, but if it's not too much trouble to use the aws cli, it's probably the easiest way to ensure the basic aws configuration is correct.

I've tried this and I can cp using aws from a dataset that I synced previously using rclone to a new remote. This dataset is not retrievable on dvc on the new remote.

[dberenbaum]

Thanks @FeryET! It seems like AWS is not the issue then.

The easiest solution if possible might be to migrate the data again using the method suggested in https://stackoverflow.com/questions/67635688/installation-dvc-on-minio-storage.

If that's not feasible, could you also try to download the data using mc cp to check if there's a MinIO issue?

[FeryET]

Thanks @FeryET! It seems like AWS is not the issue then.

The easiest solution if possible might be to migrate the data again using the method suggested in https://stackoverflow.com/questions/67635688/installation-dvc-on-minio-storage.

If that's not feasible, could you also try to download the data using mc cp to check if there's a MinIO issue?

Unfortunately, that is my last resort because the dataset I've migrated is around 1T and it's hard to download it locally and upload it again.

I cannot use mc cp either. The error is the same as what DVC has shown me:

There were headers present in the request which were not signed.

Is there anything I can do to make my minio bucket usable at this rate?

[dberenbaum]

In that case, the good news is that it seems we have isolated the error to some MinIO issue. The bad news is that I don't know enough about MinIO to help much. It doesn't seem it has anything to do with DVC. Would you be able to open an issue or ask for help from MinIO folks about why you can't mc cp the data that you copied via rclone?