将大文件上载到Amazon S3时出现问题
我尝试使用Amazon-SDK（Java）示例代码S3TransferProgressSample.java将大文件上传到Amazon-S3存储（ 也在AWS docs上发布 ）。
但是当我尝试上传11 GB文件时，上传卡在不同的位置，并显示错误消息：
Unable to upload file to Amazon S3: Unable to upload part: Unable toexecute HTTP request: Unbuffered entity enclosing request can not be repeated " (attached screenshot).
看起来在IOException发生后，SDK无法重试请求（见下文）。
有没有人遇到这个？ 解决这个问题的最佳做法是什么？ 任何代码都表示赞赏。
INFO: Received successful response: 200, AWS Request ID: 2B66E7669E24DA75
Jan 15, 2011 6:44:46 AM com.amazonaws.http.HttpClient execute
INFO: Sending Request: PUT s3.amazonaws.com /test_file_upload/autogenerated.txt Parameters: (uploadId: m9MqxzD484Ys1nifnX._IzJBGbCFIoT_zBg0xdd6kkZ4TAtmcG0lXQOE.LeiSEuqn6NjcosIQLXJeKzSnKllmw--, partNumber: 1494, )
Jan 15, 2011 6:45:10 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
**INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**
Jan 15, 2011 6:45:10 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
Jan 15, 2011 6:45:12 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:12 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
**INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**
Jan 15, 2011 6:45:12 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
Jan 15, 2011 6:45:13 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
**INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**
Jan 15, 2011 6:45:13 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
Jan 15, 2011 6:45:13 AM com.amazonaws.http.HttpClient execute
**WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.**
Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:15 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:16 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:16 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:17 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient execute
WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.
Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient execute
....
Jan 15, 2011 6:45:21 AM com.amazonaws.http.HttpClient handleResponse
**INFO: Received successful response: 204, AWS Request ID: E794B8FCA4C3D007**
Jan 15, 2011 6:45:21 AM com.amazonaws.http.HttpClient execute
...
Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient execute
INFO: Sending Request: DELETE s3.amazonaws.com /test_file_upload/autogenerated.txt Parameters:
...
Jan 15, 2011 6:47:01 AM com.amazonaws.http.HttpClient handleErrorResponse
INFO: Received error response: Status Code: 404, AWS Request ID: 0CE25DFE767CC595, AWS Error Code: NoSuchUpload, AWS Error Message: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
尝试使用低级API 。
当出现问题时，这将为您提供更多控制，因为它们可能与11GB文件有关。
来往S3的请求确实会失败。 使用低级API，如果失败，您将能够重试上传的一部分。
重新编写亚马逊文档中的示例：
// Step 2: Upload parts. long filePosition = 0; for (int i = 1; filePosition < contentLength; i++) { // Last part can be less than 5 MB. Adjust part size. partSize = Math.min(partSize, (contentLength - filePosition)); // Create request to upload a part. UploadPartRequest uploadRequest = new UploadPartRequest() .withBucketName(existingBucketName).withKey(keyName) .withUploadId(initResponse.getUploadId()).withPartNumber(i) .withFileOffset(filePosition) .withFile(file) .withPartSize(partSize); // repeat the upload until it succeeds. boolean anotherPass; do { anotherPass = false; // assume everythings ok try { // Upload part and add response to our list. partETags.add(s3Client.uploadPart(uploadRequest).getPartETag()); } catch (Exception e) { anotherPass = true; // repeat } } while (anotherPass); filePosition += partSize; } // Step 3: complete. CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest( existingBucketName, keyName, initResponse.getUploadId(), partETags); s3Client.completeMultipartUpload(compRequest);
注意：我不是一个java开发人员，所以我可以在语法上弄乱一些东西，但希望这能让你朝着正确的方向前进。 此外，如果上传重复失败，您还需要添加“重试计数器”以防止无限循环。
我认为你应该尝试AWS支持的Multipart API。
看看这个： http ： //docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html
作为旁注，如果您尝试将多部分上载到已经在分段上传下的密钥，则可能会抛出404错误。
您是否尝试上传大小为11GB的单个文件？ 或者你所有文件的大小是11GB？ 因为S3上的最大文件大小限制是5GB。
Geoff Appleford的答案对我有用。 但是，我会在while循环控制语句中添加&& retryCount 阿维亚德
我想对Geoff Appleford的答案添加评论但是SO不允许我这样做。 一般来说，他使用低级API的答案工作正常，但即使我们现在有一个do-while循环，循环的设计方式还有内置的重试逻辑。 在他的代码片段中，文件位置仅在成功时才会增加，否则您将再次上传相同的部分。