Welcome to ShenZhenJia Knowledge Sharing Community for programmer and developer-Open, Learning and Share
menu search
person
editAsk a Question
Welcome To Ask or Share your Answers For Others

Categories

java - JVM unable to create a thread in Cloud Foundry

in Technique[技术] by (71.8m points)

I'm using Spring boot for the application. There is a particular controller when called, makes a couple of independent requests using OkHttpClient. I am using a thread pool of 200 threads to make these calls. It works absolutely fine in the local environment. However, when deployed to cloud foundry, it shows up an error as follows.

I have reduced the number of threads to 100 but the problem still remains. These are the memory settings in my CF instance -

Instance Memory Limit  8192 MB
Memory Quota  3072 MB (available memory 1024 MB)
Disk Quota  1024 MB

This is the error - 

 2019-05-08T19:13:35.68+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.686  INFO 6 --- [ool-2-thread-65] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:35.76+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.769  INFO 6 --- [ool-2-thread-62] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:35.81+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.813  INFO 6 --- [ool-2-thread-54] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:35.95+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.959  INFO 6 --- [ool-2-thread-73] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:35.96+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.967  INFO 6 --- [ool-2-thread-60] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:35.96+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.967  INFO 6 --- [ool-2-thread-79] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.03+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.034  INFO 6 --- [ool-2-thread-77] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.05+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.054  INFO 6 --- [ool-2-thread-75] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.06+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.066  INFO 6 --- [ool-2-thread-18] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.11+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.119  INFO 6 --- [ool-2-thread-12] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.121  INFO 6 --- [ool-2-thread-32] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.123  INFO 6 --- [ool-2-thread-71] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] ERR Resource exhaustion event: the JVM was unable to create a thread.
   2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] ERR ResourceExhausted! (1/0)
   2019-05-08T19:13:36.13+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.136  INFO 6 --- [ool-2-thread-44] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | Instance Count | Total Bytes | Class Name                                                                                                       |
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.583  INFO 6 --- [ool-2-thread-92] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.583  INFO 6 --- [ool-2-thread-59] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.583  INFO 6 --- [ool-2-thread-38] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.584  INFO 6 --- [ool-2-thread-25] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 100796         | 11914496    | [C                                                                                                               |
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.584  INFO 6 --- [ool-2-thread-78] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.585  INFO 6 --- [ool-2-thread-67] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.585  INFO 6 --- [ol-2-thread-100] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 99630          | 2391120     | Ljava/lang/String;                                                                                               |
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.585  INFO 6 --- [ool-2-thread-80] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.586  INFO 6 --- [ool-2-thread-10] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 6940           | 2320424     | [B                                                                                                               |
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 41544          | 1661760     | Ljava/util/LinkedHashMap$Entry;                                                                                  |
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 50549          | 1617568     | Ljava/util/HashMap$Node;                                                                                         |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 13832          | 1535128     | Ljava/lang/Class;                                                                                                |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 42807          | 1369824     | Ljava/util/concurrent/ConcurrentHashMap$Node;                                                                    |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 22775          | 1279152     | [Ljava/util/HashMap$Node;                                                                                        |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 21141          | 1183896     | Ljava/util/LinkedHashMap;                                                                                        |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 10387          | 914056      | Ljava/lang/reflect/Method;                                                                                       |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 11925          | 913696      | [Ljava/lang/Object;                                                                                              |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 2202           | 618392      | [I                                                                                                               |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 36548          | 584768      | Ljava/lang/Object;                                                                                               |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 1168           | 485888      | Lcom/sap/LMChatbot/services/metricSearch;                                                                        |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 992            | 372992      | Ljava/lang/Thread;                                                                                               |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 200            | 370440      | [Ljava/util/concurrent/ConcurrentHashMap$Node;                                                                   |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 18205          | 291280      | Ljava/util/LinkedHashSet;                                                                                        |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 1              | 262160      | [Lorg/h2/mvstore/MVMap;                                                                                          |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 8144           | 195456      | Ljava/util/ArrayList;                                                                                            |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 2277           | 182544      | [Ljava/lang/ThreadLocal$ThreadLocalMap$Entry;                                                                    |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 7532           | 172640      | [Ljava/lang/Class;                                                                                               |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 2964           | 142272      | Lorg/json/simple/JSONObject;                                                                                     |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 2177           | 121576      | [Ljava/security/ProtectionDomain;                                                                                |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2406           | 114776      | [Ljava/lang/String;                                                                                              |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 4127           | 99048       | Lorg/springframework/core/MethodClassKey;                                                                        |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 1986           | 95328       | Ljava/util/HashMap;                                                                                              |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 845            | 94640       | Ljava/net/SocksSocketImpl;                                                                                       |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2319           | 92760       | Ljava/util/TreeMap$Entry;                                                                                        |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 1648           | 92288       | Lokio/Okio$4;                                                                                                    |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2185           | 87400       | Ljava/security/AccessControlContext;                                                                             |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 3042           | 73008       | Ljava/util/Collections$UnmodifiableRandomAccessList;                                                             |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2242           | 71744       | Ljava/util/LinkedList;                                                                                           |
   2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 2612           | 62688       | Ljava/util/LinkedList$Node;                                                                                      |
   2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 826            | 59472       | Lokhttp3/internal/connection/RealConnection;                                                                     |
   2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 1846           | 59072       | Ljava/util/concurrent/locks/ReentrantLock$NonfairSync;

与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…
thumb_up_alt 0 like thumb_down_alt 0 dislike
163 views
Welcome To Ask or Share your Answers For Others

1 Answer

by (71.8m points)

So, each of these threads make a REST API call. I was creating a new instance of OkHttpClient for each thread. As it turns out, each client instance would spawn many threads of its own internally. So the total number of threads were more than what I specified.

After sharing the same instance for the OkHttpClient client with all the threads, the issue seems to be resolved.


与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…
thumb_up_alt 0 like thumb_down_alt 0 dislike
Welcome to ShenZhenJia Knowledge Sharing Community for programmer and developer-Open, Learning and Share

Just Browsing Browsing

548k questions

547k answers

4 comments

86.3k users

Most popular tags

Theme made by Momin Raza, Modified by Ostack
Powered by Question2Answer
...