Question
6
Replies
2310
Views
Cognizant
Posted: January 30, 2019
Last activity: May 13, 2019
Closed
ERROR - I/O Exception occurred while writing or flushing the stream
We ran a load test for almost 15 hours. It ran fine till 12 hours. After that we started getting below errors. There were few hiccups in between also but system recovered but once these errors started system never recovered. Searched a lot but didn't get any concrete reason. Attached PegaRules logs for reference
2019-01-30 06:18:16,969 [http-executor-threads - 228] [] [] [] (web.impl.WebStandardImpl) ERROR - I/O Exception occurred while writing or flushing the stream ;
ClientAbortException: java.net.SocketException: Software caused connection abort: socket write error;
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:429);
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450);
2019-01-30 06:18:16,969 [http-executor-threads - 228] [] [] [] (web.impl.WebStandardImpl) ERROR - I/O Exception occurred while closing the Stream ;
ClientAbortException: java.net.SocketException: Software caused connection abort: socket write error;
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:429);
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450);
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:474);
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:441);
***Edited by Moderator Marissa to update platform capability tags****
Posted: 1 year ago
Thanks Arvind but the link didn't provide much clue on the issue. The load was always on the system for 12 hours but not sure why it started getting those errors at that particular time itself and subsequently the JVM crashed with memory full error.
Posted: 1 year ago
I did not review this OOM and could see this now in logs.
OOM is fallout of larger requestors on node ( 522 at one time) which is too heavy load on 8GB heap.
I suggest increasing number of nodes in the cluster.
Posted: 1 year ago
Thanks Arvind. Is there any way we can evaluate what is optimum number of requestors for a particular heap size. For example:- 200 requestors are ideal for 8 GB heap. How we can evaluate this ratio of "No. of requestors : Heap Size"
Posted: 1 year ago
Facing the same issue "I/O Exception occurred while writing or flushing the stream", any resolution for it?
Posted: 1 year ago
We are also facing the same issue .Any resolution for it .
I have opened other thread for the same issue
https://community1.pega.com/community/pega-support/question/dev-slowness-due-pega-thread-dump
2019-05-13 10:25:54,266 [.100.156-443-exec-11] [ ] [ ] [ ] ( web.impl.WebStandardImpl) WARN - I/O Exception occurred while closing the Stream
org.apache.catalina.connector.ClientAbortException: java.io.IOException: An established connection was aborted by the software in your host machine
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:356) ~[catalina.jar:8.5.30]
at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:815) ~[catalina.jar:8.5.30]
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:310) ~[catalina.jar:8.5.30]
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:263) ~[catalina.jar:8.5.30]
at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:157) ~[catalina.jar:8.5.30]
at com.pega.jcraft.jzlib.DeflaterOutputStream.close(DeflaterOutputStream.java:122) ~[prjzlib-1.1.0.jar:?]
at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:320) ~[?:1.8.0_202]
at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149) ~[?:1.8.0_202]
at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233) ~[?:1.8.0_202]
at com.pega.pegarules.web.impl.DirectStreamWriter.close(DirectStreamWriter.java:179) ~[prwebj2ee.jar:?]
at com.pega.pegarules.web.impl.WebStandardImpl.sendOutput(WebStandardImpl.java:1006) ~[prwebj2ee.jar:?]
at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:428) ~[prwebj2ee.jar:?]
at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_202]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370) ~[prbootstrap-7.3.1-218.jar:7.3.1-218]
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411) ~[prbootstrap-7.3.1-218.jar:7.3.1-218]
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:224) ~[prbootstrap-api-7.3.1-218.jar:7.3.1-218]
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:273) ~[prbootstrap-api-7.3.1-218.jar:7.3.1-218]
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebStandardBoot.java:129) ~[prbootstrap-api-7.3.1-218.jar:7.3.1-218]
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doGet(WebStandardBoot.java:100) ~[prbootstrap-api-7.3.1-218.jar:7.3.1-218]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) ~[servlet-api.jar:?]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[servlet-api.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[catalina.jar:8.5.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.30]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-websocket.jar:8.5.30]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[catalina.jar:8.5.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.30]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[catalina.jar:8.5.30]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[catalina.jar:8.5.30]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) ~[catalina.jar:8.5.30]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) ~[catalina.jar:8.5.30]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) ~[catalina.jar:8.5.30]
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) ~[catalina.jar:8.5.30]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) ~[catalina.jar:8.5.30]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) ~[catalina.jar:8.5.30]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) ~[tomcat-coyote.jar:8.5.30]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) ~[tomcat-coyote.jar:8.5.30]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) ~[tomcat-coyote.jar:8.5.30]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) ~[tomcat-coyote.jar:8.5.30]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-coyote.jar:8.5.30]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-util.jar:8.5.30]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Caused by: java.io.IOException: An established connection was aborted by the software in your host machine
at sun.nio.ch.SocketDispatcher.write0(Native Method) ~[?:1.8.0_202]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51) ~[?:1.8.0_202]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_202]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:1.8.0_202]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_202]
at org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioChannel.java:144) ~[tomcat-coyote.jar:8.5.30]
at org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:659) ~[tomcat-coyote.jar:8.5.30]
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101) ~[tomcat-coyote.jar:8.5.30]
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157) ~[tomcat-coyote.jar:8.5.30]
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1267) ~[tomcat-coyote.jar:8.5.30]
at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:670) ~[tomcat-coyote.jar:8.5.30]
at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:450) ~[tomcat-coyote.jar:8.5.30]
at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:388) ~[tomcat-coyote.jar:8.5.30]
at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:623) ~[tomcat-coyote.jar:8.5.30]
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:121) ~[tomcat-coyote.jar:8.5.30]
at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:225) ~[tomcat-coyote.jar:8.5.30]
at org.apache.coyote.Response.doWrite(Response.java:541) ~[tomcat-coyote.jar:8.5.30]
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351) ~[catalina.jar:8.5.30]
... 44 more
This could a known issue when there is a load on system:
https://it.toolbox.com/question/javanetsocketexception-software-caused-connection-abort-socket-write-error-081016