Wrong md5 comparison in case of multipart upload

[alex-pekurovsky]

If file was uploaded to bucket using multipart upload (which is nowadays default behavior of most of SDK), it's ETag is not md5 of the file. And s3_file provider compares md5 of local file with ETag. Currently, as workaround, I'm reuploading files with older PUT method, however need some procedure to support multipart uploaded files.

[mattlqx]

Further, if an object was uploaded with encryption, it won't have an ETag that's an MD5 of the file either.

From http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html:

Objects created by the PUT Object, POST Object, or Copy operation, or through the AWS Management Console, and are encrypted by SSE-C or SSE-KMS, have ETags that are not an MD5 digest of their object data.

Using the ETag against the local file content isn't a reliable method to determine if the file has changed. Amazon has simply overloaded that header too much and provides no other determinators in the header to differentiate between the format.

I'm thinking the best solution would be to keep a catalog of ETags when s3_file downloads something and use that on subsequent runs to determine if there is a change, then it's apples-to-apples and you don't have to worry about what information the ETag is derived from.

[joekiller]

Merged the fix. please test.