Ideal candidate: skilled python developer with solid knowledge of cloud and distributed systems.
Create a python application that uploads a set of given files to a cloud object storage in parallel through the cloud provider's or third party API.
- Support up to 100,000nds of files, all inside one directory with arbitrary sizes. The root directory may contain subdirectories.
- The object storage container which holds the objects is private and only credential-based access is allowed.
- Each object inside object storage should have an associated metadata which contains file size, last modification time and file permissions.
- Fast (utilize full network bandwidth), low CPU (do not block all other processes) and low Memory (<25% tentatively) file uploader
- Support for AWS S3
- Modular and Object oriented implementation (to add other cloud providers)
- Clean and documented code
- Tests
We leave exact timing to the candidate. Must fit Within 5 days total.
- we can provide temporary credentials to access AWS/Azure.