Java OKHTTP SocketTimeoutException with concurrent multithreaded requests

Question

My Java application uses a variety of proxy servers to collect data from a particular domain. The special application requires the following procedure:

Loading a URL through a specific proxy
Wait 5 seconds
Loading the next url through the same proxy

So that the loading of the information (due to the 5 second pause) doesn't take forever, I work with a total of 400 threads. Each of these threads uses its own proxy server, i.e. also with its own OKHTTP client:

MyHTTPClient = new OkHttpClient.Builder()
.connectTimeout(7, TimeUnit.SECONDS)
.writeTimeout(7, TimeUnit.SECONDS)
.readTimeout(7, TimeUnit.SECONDS)
.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ProxyIP, ProxyPort)))
.proxyAuthenticator((Route route, Response response) - > {
    return response.request().newBuilder().header("Proxy-Authorization", Credentials.basic(ProxyUser, ProxyPass)).build();
})
.sslSocketFactory(sslSocketFactory, (X509TrustManager) trustAllCerts[0])
.hostnameVerifier((String hostname, SSLSession session) - > true)
.addNetworkInterceptor(new LoggingInterceptor())
.retryOnConnectionFailure(true)
.build();

Each thread has to use its own proxy, so each thread has its own OKHTTP client. In total there are 400 OKHTTP clients.

Each thread now processes its queries as follows:

 while (true) {
     MyAnswer = MyHTTPClient.newCall(
         new Request.Builder().url(https://www.example.com)
         .addHeader("referer", SomeReferrer)
         .addHeader("cache-control", "no-cache")
         .addHeader("pragma", "no-cache")
         .get().build())
     .execute();

     Body = MyAnswer.body().string();
     MyAnswer.body().close();

     Thread.sleep(5000);
 }

And that works great too, for about a minute. Then suddenly the following error messages appear for each individual request on every single thread:

java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.waitForConnect(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:107)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.base/java.net.Socket.connect(Socket.java:608)
at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:120)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
at okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.kt:261)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:201)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at okio.InputStreamSource.read(JvmOkio.kt:90)
at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:129)
... 22 more

WHAT I TRIED SO FAR

I've done a bit of research and found out that there might be some kind of limit in terms of concurrent connections. I found out, that whenever this error occurs, there are no idle connections in the connection pool:

MyHTTPClient.connectionPool().idleConnectionCount() //Always 0 when the Timeout-Exception occurs

So i tried to increase the size of the connection pool for every single okhttpclient but that seemed to have absolutely no effect:

.connectionPool(new ConnectionPool(5000, 100, TimeUnit.MILLISECONDS))

I also tried to increase and decrease the timeout-values of the httpclient and that actually seemed to make a difference in terms of how long the application would run without this error but it actually didn't prevent this to happen.

I also tried to increase the amount of simultaneous connections and decrease the amount of time a connection stays in "TIME_WAIT" status as mentioned here: https://learn.microsoft.com/en-us/answers/questions/482793/tcpip-cuncurrent-connections.html

I also tried to add a Dispatcher to the OKHTTPClient, as follows:

Dispatcher MyDispatcher = new Dispatcher(Executors.newCachedThreadPool());
MyDispatcher.setMaxRequests(9999999);
MyDispatcher.setMaxRequestsPerHost(9999999);

If you have any ideas about what could be the cause or what else I could try, please let me know. I am also happy to send you further information, if needed.

Thank you in advance!

Do you have access to the server? It seems possible that you get throttled or that the server simply can't handle that many simultaneous requests. Typically you would setup a firewall to block an IP if it sends too many requests in a given interval. — rmfeldt
– rmfeldt, Commented Oct 12, 2021 at 10:40
@rmfeldt Yes I do. I noticed that on the same computer, the network is generally getting quite slow but on another computer on the same network, everything works fine and the responose times of the queried website are still absolutely incredibly fast. So there shouldn't be a problem with throtteling from the website itself. — MMA
– MMA, Commented Oct 12, 2021 at 10:45

Jesse Wilson · Accepted Answer · 2021-10-12 11:30:09Z

2

Create one OkHttpClient initially and then use OkHttpClient.newBuilder() to customize the proxy. This will cause the clients to share thread pools and connection pools. That might not fix your problem on its own, but it'll make it more efficient.

  final OkHttpClient sharedHttpClient = new OkHttpClient.Builder()
    .connectTimeout(7, TimeUnit.SECONDS)
    .writeTimeout(7, TimeUnit.SECONDS)
    .readTimeout(7, TimeUnit.SECONDS)
    .sslSocketFactory(sslSocketFactory, (X509TrustManager) trustAllCerts[0])
    .hostnameVerifier((String hostname, SSLSession session) - > true)
    .addNetworkInterceptor(new LoggingInterceptor())
    .retryOnConnectionFailure(true)
    .build();

MyHTTPClient = sharedHttpClient.newBuilder()
    .proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ProxyIP, ProxyPort)))
    .proxyAuthenticator((Route route, Response response) - > {
    return response.request().newBuilder().header("Proxy-Authorization", Credentials.basic(ProxyUser, ProxyPass)).build();
})
    .build();

answered Oct 12, 2021 at 11:30

Jesse Wilson

40.8k8 gold badges132 silver badges134 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

MMA Over a year ago

Thank you very much, I just implemented this! But sadly you are right, that doesn't solve the actual problem.

djzhu · Accepted Answer · 2022-01-05 03:03:22Z

I have met a similar situation last day. Here are my ideas about these problem:

The first step is to find out which kind of timeout you have occurred. Since there are several types of timeout scene in httpclient. Such as:
- timeout to fetch a connection from connectionPool
- timeout to establish a socket connection
- timeout to do wite and read operation

I have a little trick to accomplish this goal: Change the settings of timeout in OkHttpClient.Builder(), let different kind of timeout vary from each one others. And then print the actual timeout duration in log, so you can find out which one it truely match.

Check if there is any abnormal tcp connection. Since you have use connection pool in your program, it may not mantains too much connections at the same time. You can check the number of connection by:

# each state of connection
netstat -n| awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
# connection to specifed port
netstat -ntap|grep :PORT

Check the response time in server side. Depends on your web container, you can see if there is any request take too long to return. But it may not be your answer, since you have confirm another computor doesn't have the same issue.
Check the network itself. Use this script to check if any package take too long(-w param specified the timeout duration) to return:

for i in {1..10000}; do nc -vnz IP PORT -w 1 2>&1 |grep "timed out"; done

That's all, hope that may help you.

Collectives™ on Stack Overflow

Java OKHTTP SocketTimeoutException with concurrent multithreaded requests

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related