Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Infer storage_type from file name (always?) #93

Open
valeriupredoi opened this issue Jun 15, 2023 · 2 comments
Open

Infer storage_type from file name (always?) #93

valeriupredoi opened this issue Jun 15, 2023 · 2 comments

Comments

@valeriupredoi
Copy link
Collaborator

In #89 I had to introduce a new kwarg to Active -> storage_type to have correct open/retrieve operations depending on the type of storage (POSIX vs S3 now) - storage_type should/could be picked up from file name as @davidhassell suggested offline - but we can change that later (am not 100% sure we should, though, since future possible storage types may not necessarily be guessed from file name) - this issue is to raise our awareness as we go along.

@markgoddard
Copy link

You might want to reconsider how the S3 access is configured - is global config in config.py the most appropriate approach? I used it as a fast way to get things working.

Would it be better to pass in the URLs & credentials etc. to the Active object's constructor? Or possibly abstract them into a Storage object that provides a reduce_chunk interface and "knows" how to access the object in storage.

@valeriupredoi
Copy link
Collaborator Author

that's a very good point and question! I am thinking we should probably discuss what other types of storage we want to support, and then see what's the best way to have them configured at runtime - I was too thinking about config.py when you opened the PR a few weeks ago and was leaning towards a yaml config file (still sort of an externally loaded configuration) but a config object might be a better approach. For now it does the trick as interim solution 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants