从S3下载文件> 3Gb失败,出现“SocketTimeoutException:Read timed out”

AWS Java SDK 1.9.3

在Java服务器应用程序中从AWS S3下载许多大文件(~3Gb)时,我会不时收到SocketTimeoutException,如下所示:

Caused by: com.amazonaws.AmazonClientException: Unable to store object contents to disk: Read timed out at com.amazonaws.services.s3.internal.ServiceUtils.downloadObjectToFile(ServiceUtils.java:270) at com.amazonaws.services.s3.internal.ServiceUtils.retryableDownloadS3ObjectToFile(ServiceUtils.java:344) at com.amazonaws.services.s3.transfer.TransferManager$2.call(TransferManager.java:737) ... 4 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554) at sun.security.ssl.InputRecord.read(InputRecord.java:509) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:198) at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73) at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73) at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151) at java.security.DigestInputStream.read(DigestInputStream.java:161) at com.amazonaws.services.s3.internal.DigestValidationInputStream.read(DigestValidationInputStream.java:59) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73) at java.io.FilterInputStream.read(FilterInputStream.java:107) at com.amazonaws.services.s3.internal.ServiceUtils.downloadObjectToFile(ServiceUtils.java:265) ... 6 more 

我使用4个线程从S3下载文件,所有线程都使用相同的TransferManager实例。 如前所述,我正在使用AWS SDK 1.9.3并且只是不时获得此类例外。 代码如下:

 ... Download d = transferManager.download(currentBucket, remoteFileName, new File(tmpName)); d.waitForCompletion(); ... 

这是SDK的问题吗? 也许是v1.9.3?

除了增加ClientConfiguration socketTimeout之外,还有其他解决方案吗?

以下是我在AWS SDK 1.9.3中的操作方法:

 ClientConfiguration config = new ClientConfiguration(); config.setConnectionTimeout(connectionTimeout); config.setSocketTimeout(readTimeout); AmazonS3 s3 = new AmazonS3Client(credentials, config);