Question
ERROR: Retrieve CSV file / data from HDFS File System
Hi
If I save a csv file from Pega into Hadoop - HDFS file system, I am able to retrieve the file but I am getting the below error while I am trying to retrieve any other csv file (not saved/created from Pega in HDFS) from Hadoop HDFS file system (in Cloudera distribution).
Could not obtain block: BP-171906535-127.0.0.1-1491391569011:blk_1073742887_2063 file=/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/Test1-00002.csv
Complete Error Message:
18:26:33.072 [http-nio-8080-exec-9] WARN org.apache.hadoop.hdfs.DFSClient - Cou
ld not obtain block: BP-171906535-127.0.0.1-1491391569011:blk_1073742887_2063 fi
le=/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/Test1-00002.csv No live node
s contain current block Block locations: 10.0.2.15:50010 Dead nodes: 10.0.2.15:
50010. Throwing a BlockMissingException
18:26:33.072 [http-nio-8080-exec-9] WARN org.apache.hadoop.hdfs.DFSClient - Cou
ld not obtain block: BP-171906535-127.0.0.1-1491391569011:blk_1073742887_2063 fi
le=/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/Test1-00002.csv No live node
s contain current block Block locations: 10.0.2.15:50010 Dead nodes: 10.0.2.15:
50010. Throwing a BlockMissingException
18:26:33.072 [http-nio-8080-exec-9] WARN org.apache.hadoop.hdfs.DFSClient - DFS
Read
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1719065
35-127.0.0.1-1491391569011:blk_1073742887_2063 file=/var/lib/hadoop-hdfs/cache/m
apred/mapred/staging/Test1-00002.csv
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.j
ava:880) ~[hadoop-hdfs-2.4.1.jar:?]
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java
:560) ~[hadoop-hdfs-2.4.1.jar:?]
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream
.java:790) [hadoop-hdfs-2.4.1.jar:?]
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:837) [
hadoop-hdfs-2.4.1.jar:?]
at java.io.DataInputStream.read(DataInputStream.java:149) [?:1.8.0_111]
at com.pega.bigdata.dataset.hdfs.HDFSFileClient$ReadFileContentHDFSOpera
tion.readFromStream(HDFSFileClient.java:228) [bigdata-platform-7.2.2.jar:?]
at com.pega.bigdata.dataset.hdfs.HDFSFileClient$ReadFileContentHDFSOpera
tion.perform(HDFSFileClient.java:196) [bigdata-platform-7.2.2.jar:?]
at com.pega.bigdata.dataset.hdfs.HDFSFileClient$ReadFileContentHDFSOpera
tion.perform(HDFSFileClient.java:168) [bigdata-platform-7.2.2.jar:?]
at com.pega.bigdata.dataset.hdfs.HDFSFileClient$SingleHDFSOperation.perf
orm(HDFSFileClient.java:1046) [bigdata-platform-7.2.2.jar:?]
at com.pega.bigdata.hdfs.HDFSExecutor.performOnCachedFileSystem(HDFSExec
utor.java:150) [bigdata-platform-7.2.2.jar:?]
at com.pega.bigdata.hdfs.HDFSExecutor.performOperation(HDFSExecutor.java
:89) [bigdata-platform-7.2.2.jar:?]
at com.pega.bigdata.dataset.hdfs.HDFSFileClient.getFileContent(HDFSFileC
lient.java:393) [bigdata-platform-7.2.2.jar:?]
at com.pegarules.generated.activity.ra_action_pzpreviewfile_874845c432dc
ce16eb4dce06edf20a1d.step7_circum0(ra_action_pzpreviewfile_874845c432dcce16eb4dc
e06edf20a1d.java:722) [prgenclasses.jar:?]
at com.pegarules.generated.activity.ra_action_pzpreviewfile_874845c432dc
ce16eb4dce06edf20a1d.perform(ra_action_pzpreviewfile_874845c432dcce16eb4dce06edf
20a1d.java:184) [prgenclasses.jar:?]
at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Execut
able.java:3553) [prprivate.jar:?]
at com.pegarules.generated.activity.ra_action_pzrunactionwrapper_056ec03
796d8a13802f156886066036d.step1_circum0(ra_action_pzrunactionwrapper_056ec03796d
8a13802f156886066036d.java:319) [prgenclasses.jar:?]
at com.pegarules.generated.activity.ra_action_pzrunactionwrapper_056ec03
796d8a13802f156886066036d.perform(ra_action_pzrunactionwrapper_056ec03796d8a1380
2f156886066036d.java:70) [prgenclasses.jar:?]
at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Execut
able.java:3553) [prprivate.jar:?]
at com.pega.pegarules.session.internal.mgmt.base.ThreadRunner.runActivit
iesAlt(ThreadRunner.java:646) [prprivate.jar:?]
at com.pega.pegarules.session.internal.mgmt.PRThreadImpl.runActivitiesAl
t(PRThreadImpl.java:465) [prprivate.jar:?]
at com.pega.pegarules.session.internal.engineinterface.service.HttpAPI.r
unActivities(HttpAPI.java:3203) [prprivate.jar:?]
at com.pega.pegarules.session.external.engineinterface.service.EngineAPI
.processRequestInner(EngineAPI.java:391) [prenginext.jar:?]
at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43) ~[?:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTarg
etActionWithLock(PRSessionProviderImpl.java:1297) [prprivate.jar:?]
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithReque
storLocked(PRSessionProviderImpl.java:1034) [prprivate.jar:?]
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithReque
storLocked(PRSessionProviderImpl.java:889) [prprivate.jar:?]
at com.pega.pegarules.session.external.engineinterface.service.EngineAPI
.processRequest(EngineAPI.java:331) [prenginext.jar:?]
at com.pega.pegarules.session.internal.engineinterface.service.HttpAPI.i
nvoke(HttpAPI.java:829) [prprivate.jar:?]
at com.pega.pegarules.session.internal.engineinterface.etier.impl.Engine
Impl._invokeEngine_privact(EngineImpl.java:331) [prprivate.jar:?]
at com.pega.pegarules.session.internal.engineinterface.etier.impl.Engine
Impl.invokeEngine(EngineImpl.java:274) [prprivate.jar:?]
at com.pega.pegarules.session.internal.engineinterface.etier.impl.Engine
Impl.invokeEngine(EngineImpl.java:251) [prprivate.jar:?]
at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngineInner(JND
IEnvironment.java:278) [prpublic.jar:?]
at com.pega.pegarules.priv.context.JNDIEnvironment.invokeEngine(JNDIEnvi
ronment.java:223) [prpublic.jar:?]
at com.pega.pegarules.web.impl.WebStandardImpl.makeEtierRequest(WebStand
ardImpl.java:589) [prwebj2ee.jar:?]
at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.ja
va:378) [prwebj2ee.jar:?]
at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43) ~[?:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBoot
strap.java:370) [prbootstrap-7.2.2-117.jar:7.2.2-117]
at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropaga
tingThrowable(PRBootstrap.java:411) [prbootstrap-7.2.2-117.jar:7.2.2-117]
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invo
keMethodPropagatingThrowable(AppServerBridgeToPega.java:224) [prbootstrap-api-7.
2.2-117.jar:7.2.2-117]
at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invo
keMethod(AppServerBridgeToPega.java:273) [prbootstrap-api-7.2.2-117.jar:7.2.2-11
7]
at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebSta
ndardBoot.java:121) [prbootstrap-api-7.2.2-117.jar:7.2.2-117]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) [servlet
-api.jar:?]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [servlet
-api.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:230) [catalina.jar:8.5.8]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:165) [catalina.jar:8.5.8]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52
) [tomcat-websocket.jar:8.5.8]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:192) [catalina.jar:8.5.8]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:165) [catalina.jar:8.5.8]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:198) [catalina.jar:8.5.8]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:108) [catalina.jar:8.5.8]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authentica
torBase.java:472) [catalina.jar:8.5.8]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:140) [catalina.jar:8.5.8]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
ava:79) [catalina.jar:8.5.8]
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAcce
ssLogValve.java:620) [catalina.jar:8.5.8]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
ve.java:87) [catalina.jar:8.5.8]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
a:349) [catalina.jar:8.5.8]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java
:783) [tomcat-coyote.jar:8.5.8]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLig
ht.java:66) [tomcat-coyote.jar:8.5.8]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(Abstract
Protocol.java:789) [tomcat-coyote.jar:8.5.8]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpo
int.java:1455) [tomcat-coyote.jar:8.5.8]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBas
e.java:49) [tomcat-coyote.jar:8.5.8]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:617) [?:1.8.0_111]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskTh
read.java:61) [tomcat-util.jar:8.5.8]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
2017-09-12 18:26:33,076 [http-nio-8080-exec-9] [TABTHREAD0] [
] [ BankerApp:01.01.01] (Data_Admin_DataSet_HDFS.Action) ERROR inp-51xk882|169
Hi,
Not very sure but looks like your Name Node cannot find the blocks. Can you try restarting node managers on all datanodes and see if it resolves your issue or not.
Also can you verify If the number of datanodes in cluster and secondary namenode are showing equal number?
Thank You.