Question

2
Replies
141
Views
RanjithkumarKasula Member since 2016 2 posts
Posted: 4 years ago
Last activity: 4 years 7 months ago
Closed
Solved

Spike in WAS threads resulting in node unavailability

Issue Description: 

We have been having an issue with the JVM web container threads maxing out all of a sudden where in the threads spike up to a max of 100 from around 6 threads in couple of seconds.

Observations: 

 

  1. The JVM’s spike up instantly and runs on max threads for about 4-5 minutes and recovers on their own.
  2. Only few JVM’s are getting impacted (mostly 4 of 24 in total).
  3. There were no long running DB transactions, thread blocks; CPU utilization is well within limit(less than 60%) at the database side.
  4. This issue is not occurring consistently on the same JVM’s.
  5. This is occurring intermittently, not every day.

 

Below are the observations from our PEGA logs: 

    1. ParkNanos

 

Thread[WebContainer : 27,5,main]

 

sun.misc.Unsafe.park(Native Method)

 

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:222)

 

at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:916)

 

at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1235)

 

at com.pega.jsr166backport.java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:422)

 

at com.pega.pegarules.session.internal.mgmt.base.RequestorThreadSync.lockAttempt(RequestorThreadSync.java:403)

 

at com.pega.pegarules.session.internal.mgmt.PRRequestorImpl.lockAttempt(PRRequestorImpl.java:842)

 

at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:930)

 

at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:792)

 

at com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequest(EngineAPI.java:328)

 

at com.pega.pegarules.session.internal.engineinterface.service.HttpAPI.invoke(HttpAPI.java:703)

 

at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:312)

 

at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)

 

at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:221)

 

at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)

 

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

 

at java.lang.reflect.Method.invoke(Method.java:611)

 

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:349)

 

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:390)

 

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingException(PRBootstrap.java:412)

 

at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)

 

at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineBMT_f2439d86.invokeEngine(Unknown Source)

 

at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngineInner(JNDIEnvironment.java:277)

 

at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngine(JNDIEnvironment.java:222)

 

at com.pega.pegarules.web.impl.WebStandardImpl.makeEtierRequest(WebStandardImpl.java:373)

 

at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:266)

 

at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)

 

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

 

at java.lang.reflect.Method.invoke(Method.java:611)

 

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:349)

 

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:390)

 

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:439)

 

at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebStandardBoot.java:99)

 

at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doGet(WebStandardBoot.java:89)

 

at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)

 

at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)

 

at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1694)

 

at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:970)

 

at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:508)

 

at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)

 

at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)

 

at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:878)

 

at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)

 

at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)

 

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454)

 

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:516)

 

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:307)

 

at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)

 

at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)

 

at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)

 

at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)

 

at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)

 

at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)

 

at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:816)

 

at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)

 

at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1662)

 

 

    1. sun.misc.Unsafe.park(Native Method)

 

Thread[WebContainer : 23,5,main]

 

sun.misc.Unsafe.park(Native Method)

 

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:222)

 

at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:916)

 

at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1235)

 

at com.pega.jsr166backport.java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:422)

 

at com.pega.pegarules.session.internal.mgmt.base.RequestorThreadSync.lockAttempt(RequestorThreadSync.java:403)

 

at com.pega.pegarules.session.internal.mgmt.PRRequestorImpl.lockAttempt(PRRequestorImpl.java:842)

 

at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:930)

 

at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:792)

 

at com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequest(EngineAPI.java:328)

 

at com.pega.pegarules.session.internal.engineinterface.service.HttpAPI.invoke(HttpAPI.java:703)

 

at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(EngineImpl.java:312)

 

at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(EngineImpl.java:263)

 

at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.invokeEngine(EngineBean.java:221)

 

at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)

 

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

 

at java.lang.reflect.Method.invoke(Method.java:611)

 

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:349)

 

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:390)

 

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingException(PRBootstrap.java:412)

 

at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.invokeEngine(EngineBeanBoot.java:168)

 

at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineBMT_f2439d86.invokeEngine(Unknown Source)

 

at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngineInner(JNDIEnvironment.java:277)

 

at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngine(JNDIEnvironment.java:222)

 

at com.pega.pegarules.web.impl.WebStandardImpl.makeEtierRequest(WebStandardImpl.java:373)

 

at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:266)

 

at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)

 

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

 

at java.lang.reflect.Method.invoke(Method.java:611)

 

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:349)

 

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:390)

 

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:439)

 

at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebStandardBoot.java:99)

 

at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doGet(WebStandardBoot.java:89)

 

at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)

 

at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)

 

at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1694)

 

at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:970)

 

at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:508)

 

at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)

 

at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)

 

at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:878)

 

at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)

 

at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)

 

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454)

 

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:516)

 

at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:307)

 

at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)

 

at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)

 

at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)

 

at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)

 

at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)

 

at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)

 

at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)

 

at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)

 

at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1662)

 

 

Any thoughts to find out the RCA and resolution are very much appreciated!

 

Thanks in advance!

 

Ranjith Kasula

 

 

System Administration Data Integration
Moderation Team has archived post
Share this page LinkedIn