Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update evaporate.js #401

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Update evaporate.js #401

wants to merge 2 commits into from

Conversation

odyswise
Copy link

In our project, we faced with the possibility to upload several identical files to one s3 bucket to the separate subfolders. This cause the s3 upload error: was uploaded the only first file from the list of identical files. This patch for the fix parallel uploads of identical files.

In our project, we faced with the possibility to upload several identical files to one s3 bucket to the separate subfolders. This cause the s3 upload error: was uploaded the only first file from the list of identical files. This patch for the fix parallel uploads of identical files.
@bikeath1337 bikeath1337 self-requested a review March 13, 2018 14:59
Copy link
Collaborator

@bikeath1337 bikeath1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The available option allowS3ExistenceOptimization might work for your needs, but then you would not benefit from that optimization. Refer to the wiki for more information.

evaporate.js Outdated
@@ -2066,6 +2066,7 @@
// The key tries to give a signature to a file in the absence of its path.
// "<filename>-<mimetype>-<modifieddate>-<filesize>"
return [
fileUpload.name, // fix for parallel uploads of identical files
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this breaks compatibility and a key feature of EvaporateJS: the ability to retry failed transfers. You can see which specs fail by running the test suite.

Copy link
Author

@odyswise odyswise Mar 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I did something wrong, but in my case when I tried to upload the same file simultaneously to the one bucket to the separated directories. Evaporate's library tried to retry upload that file and I had one result in all tries - the first file upload was done successfully, all others - not. In this process, the library sends me notifications about file uploading progress and the success event. But files didn't upload.

@odyswise
Copy link
Author

I changed fileUpload.name to the fileUpload.awsKey. Maybe this will be okay for your tests? I tried use allowS3ExistenceOptimization with true and false values without success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants