Question

1
Replies
2312
Views
Saiaditya Kadapa (SaiadityaK)
Cox Communications

Cox Communications
US
SaiadityaK Member since 2017 1 post
Cox Communications
Posted: August 14, 2017
Last activity: August 16, 2017
Posted: 14 Aug 2017 13:41 EDT
Last activity: 16 Aug 2017 11:09 EDT
Closed

DNode Exception? com.pega.dsm.dnode.api.DNodeException: Cannot execute select statemen

Hello All,


we have this issue going on where we have been seeing this particular DNode exception, and when it happens we are seeing high CPU utilization, some nodes are reaching up to 300<. We had to restart the application every time. Can someone help us figure the cause. we also turned on logging level to INFO inside SMA for those particular couple of nodes. Below is the exception and some logs,


Exception:

Hello All,

we have this issue going on where we have been seeing this particular DNode exception, and when it happens we are seeing high CPU utilization, some nodes are reaching up to 300<. We had to restart the application every time. Can someone help us figure the cause. we also turned on logging level to INFO inside SMA for those particular couple of nodes. Below is the exception and some logs,

Exception:

.pega.dsm.dnode.impl.core.ExceptionWithInputRecord: com.pega.dsm.dnode.api.DNodeException: Cannot execute select statement<CR> ... 74 more<CR>Caused by: com.pega.dsm.dnode.api.DNodeException: Cannot execute select statement<CR> at com.pega.dsm.dnode.impl.cassandra.CassandraDao.executeStatement(CassandraDao.java:150)<CR> at com.pega.dsm.dnode.impl.cassandra.CassandraDataRepository.findBy(CassandraDataRepository.java:267)<CR> at com.pega.dsm.dnode.impl.dataset.cassandra.CassandraBrowseByKeysOperation$1.emit(CassandraBrowseByKeysOperation.java:56)<CR> at com.pega.dsm.dnode.impl

Logs:

100% java.lang.Thread.run():745
100% org.apache.tomcat.util.net.JIoEndpoint$Worker.run():926
100% org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(java.net.Socket):653
100% org.apache.coyote.http11.Http11Processor.process(java.net.Socket):856
100% org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request, org.apache.coyote.Response):344
100% org.apache.catalina.core.StandardEngineValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):102
100% org.apache.catalina.valves.ErrorReportValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):97
99.8% org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):145
99.8% org.jboss.as.web.security.SecurityContextAssociationValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):169
99.8% org.jboss.as.web.session.LockingValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):64
99.8% org.jboss.as.web.session.JvmRouteValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):92
99.8% org.jboss.as.web.session.ClusteredSessionValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):99
99.8% org.jboss.as.web.session.ClusteredSessionValve.handleRequest(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response, org.jboss.servlet.http.HttpEvent, boolean):134
99.8% org.apache.catalina.authenticator.AuthenticatorBase.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):420
99.8% org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):149
99.8% org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.connector.Request, org.apache.catalina.connector.Response):231
99.8% org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse):214
99.8% org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse):295
99.8% javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse):847
99.65% javax.servlet.http.HttpServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse):754
99.65% com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse):121
99.65% com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(java.lang.Object[][], int, java.lang.Object, java.lang.Object[]):272
99.65% com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(java.lang.Object[][], int, java.lang.Object, java.lang.Object[]):223
99.65% com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(java.lang.Object[][], int, java.lang.Object, java.lang.Object[]):411
99.65% com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]):370
99.65% java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]):497
99.65% sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]):43
99.65% sun.reflect.GeneratedMethodAccessor71.invoke(java.lang.Object, java.lang.Object[]):unknown
99.65% com.pega.pegarules.web.impl.WebStandardImpl.doPost(com.pega.pegarules.internal.bootstrap.interfaces.IWebStandard, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse):374
99.65% com.pega.pegarules.web.impl.WebStandardImpl.makeEtierRequest(com.pega.pegarules.internal.bootstrap.interfaces.IWebStandard, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, java.util.Map, java.lang.Object, com.pega.pegarules.web.impl.DirectStreamWriter):574
99.65% com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngine(java.lang.String, java.lang.Object):223
99.65% com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngineInner(java.lang.String, java.lang.String, java.lang.Object):278
99.65% com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(java.lang.String, java.lang.Object[]):240
99.65% com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.invokeEngine(java.lang.String, java.lang.Object[], java.util.Map):263
99.65% com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._invokeEngine_privact(java.lang.String, java.lang.Object[], java.util.Map, com.pega.pegarules.session.external.mgmt.EngineRunnable):315
82.59% com.pega.pegarules.integration.engine.internal.services.http.HTTPService.invoke(java.lang.Object[], java.util.Map):416
65.3% com.pega.pegarules.session.external.engineinterface.service.EngineAPI.processRequest():331
65.3% com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(java.lang.Object, int, long, java.lang.Object, java.lang.Object, java.lang.Object[], java.util.Map, java.lang.reflect.Method):841
65.3% com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(java.lang.Object, int, long, java.lang.Object, java.lang.Object, java.lang.Object[], java.util.Map, java.lang.Object, boolean, boolean, boolean, com.pega.pegarules.exec.external.util.TenantIdentifier):1008
65.3% com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(java.lang.Object, java.lang.Object, java.lang.Object[], java.util.Map, java.lang.Object, com.pega.pegarules.session.external.mgmt.IPRRequestor):1270
65.3% java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]):497
65.3% sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]):43
65.3% sun.reflect.GeneratedMethodAccessor72.invoke(java.lang.Object, java.lang.Object[]):unknown

More:

] (cassandra.net.MessagingService) INFO - Waiting for messaging service to quiesce

More:

Cassandra.service.GCIInspector WARN, Heap is full. You may need to reduce or memtable cache sizes.Cassandra will now flush up the two largest memtables. Adjust flush target memtables in yaml file at threshold if want to do this by cassandra automatically.

More:

DevOps
Moderation Team has archived post, This thread is closed to future replies. Content and links will no longer be updated. If you have the same/similar Question, please write a new Question.