Hi, I am trying to query about 22620 tags from a Time-series at one instance. the Micro service is running with 5G memory and 2 instances. i am using following configurations
predix.timeseries.maxTagsPerQuery=20
predix.timeseries.maxIngestionMessageSize=511000
# Predix Timeseries configuration. <<<<< 511 kbytes max
#Purchased Plan restrictions. >>>>>
# MODIFY TO SUIT YOUR PLAN. GOING OVER MIGHT RESULT IN ADDED CHARGES
plan.ingestion.concurrent.connections.max=2
plan.query.concurrent.connections.max=400
So my understanding is the library can establish max 400 connection at any time with the timeseries and can fetch only 20 tags per connection.
Sometimes Micro service gave me the following error
2017-06-22 06:58:44.726 ERROR 29 --- [io-63784-exec-9] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is **java.lang.OutOfMemoryError: unable to create new native thread] with root cause**
at java.lang.Thread.start(Thread.java:714) [na:1.8.0_101]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_101]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_101]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[tomcat-embed-core-8.5.14.jar!/:8.5.14]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar!/:8.5.14]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:81) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:349) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:392) ~[tomcat-embed-core-8.5.14.jar!/:8.5.14]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) ~[spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at java.lang.Thread.start0(Native Method) [na:1.8.0_101]
**java.lang.OutOfMemoryError: unable to create new native thread**
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.14.jar!/:8.5.14]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar!/:8.5.14]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.14.jar!/:8.5.14]
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) [tomcat-embed-core-8.5.14.jar!/:8.5.14]
I searched it on google and find out ... This is not a memory problem, but an operating system resource problem. You are running out of native threads.
how many threads the Docker Container will allow your JVM to use?
According to my understanding 400 threads are not too much, right?
Is there any way around this?
Your Help is much appreciated...!
Hi, I think this might be using the timeseries-client from the docs that doesn't give out the source code?
My team supports the timeseries-bootstrap SDK. Have you tried that one? I know that has been used successfully in many apps. That SDK is outfitted with a connection pool which would be a better way to help with thread management and memory constraints.
In your particular case, if I'm not mistaken, there could be issues with the number of open file handles or other settings that your User in your OS is allowing. In linux, ulimit -a will tell you the current settings. Something I might have an encountered a decade ago comes to mind where a thread limit was affected by this.
Answer by Thuan Truong · Jun 30, 2017 at 03:17 PM
Hi,
How do you obtain the timeseries connection? could you pls post the source code here? (how do you instance the tenantcontext and Client connection)
I'm asking that because I experienced a problem with connection management in timeseries-client lib.
Thanks,
Thuan