Skip to main content
Contact a Moderator

Question

14
Replies
1696
Views
 Johan Hoogenboezem (JohanH55) Ernst & Young Senior Manager
Ernst & Young
ZA
JohanH55 Member since 2018 11 posts
Ernst & Young
Posted: November 26, 2019
Last activity: May 21, 2020
Posted: 26 Nov 2019 3:53 EST
Last activity: 21 May 2020 18:50 EDT
Closed
Solved

Stream node remain in "JOINING_FAILED" status

Hi All

We have a cluster in Production with one node on the Stream tab of the "Decisioning: Services" landing page remaining in JOINING_FAILED status (see attached screen shot). We traced it to this snippet in the Kafka server.log file:

...snip...

Show More

Hi All

We have a cluster in Production with one node on the Stream tab of the "Decisioning: Services" landing page remaining in JOINING_FAILED status (see attached screen shot). We traced it to this snippet in the Kafka server.log file:

...snip...

[2019-11-25 11:13:23,513] INFO Creating /brokers/ids/6 (is it secure? false) (kafka.zk.KafkaZkClient)
[2019-11-25 11:13:23,541] ERROR Error while creating ephemeral at /brokers/ids/6, node already exists and owner '31139484958261249' does not match current session '31140168996945922' (kafka.zk.KafkaZkClient$CheckedEphemeral)
[2019-11-25 11:13:23,541] INFO Result of znode creation at /brokers/ids/6 is: NODEEXISTS (kafka.zk.KafkaZkClient)
[2019-11-25 11:13:23,550] ERROR [KafkaServer id=6] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1476)
at kafka.zk.KafkaZkClient.registerBrokerInZk(KafkaZkClient.scala:84)
at kafka.server.KafkaServer.startup(KafkaServer.scala:254)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:75)
at kafka.Kafka.main(Kafka.scala)
[2019-11-25 11:13:23,553] INFO [KafkaServer id=6] shutting down (kafka.server.KafkaServer)
[2019-11-25 11:13:23,555] INFO [SocketServer brokerId=6] Stopping socket server request processors (kafka.network.SocketServer)

..snip...

Does anyone have an idea of what the problem is here? We know next to nothing about Kafka.

Regards,

Johan


Show Less
Pega Customer Decision Hub 8.1 Pega Customer Service 8.1 Pega Platform 8.1.1 Case Management Decision Management Support Case Created
Moderation Team has archived post, This thread is closed to future replies. Content and links will no longer be updated. If you have the same/similar Question, please write a new Question.
Posted: 2 years ago
Posted: 27 Nov 2019 9:37 EST
Marc Lasserre (MarcLasserre_GCS) PEGA Senior Principal Engineer, Technical Support
Pegasystems Inc.
FR

Hello, 

Is it a new environment? First time you have seen this issue, how long have you been running with this Production system? any changes?
Posted: 2 years ago
Posted: 27 Nov 2019 9:57 EST
Anandh Palanisamy (Anandh) Abbvie Technology Architect II
Abbvie
US

Hi Johan,

Try to stop that JVM and check is any active process running on the app server after JVM stop.

If any kill those process and start again and also if upgraded system make sure you clean pr_sys_statusnode table.

Hope this helps, let me know how its goes.

Regards,

Anandh
Posted: 2 years ago
Posted: 28 Nov 2019 0:12 EST
Johan Hoogenboezem (JohanH55) Ernst & Young Senior Manager
Ernst & Young
ZA

Thank you for your response, Anandh. We've been having this problem since past Friday. We've been in production since June with twice-monthly deployments. There was no deployment last week. I didn't clean the pr_sys_statusnode table since we have not upgraded. Should I delete the row related to the machine with the problem? I tried your suggestion of stopping the JVM and killing any other Java processes after JVM stopped. There was one. It didn't help to kill it and restart. On the other machines there are two other Java processes, which makes sense since Kafka and Cassandra are both running fine over there.
Posted: 2 years ago
Posted: 28 Nov 2019 10:37 EST
Abhinav Gupta (Abhinav7) PEGA Manager, Technical Support - Platform Service Engineering
Pegasystems Inc.
IN

Hi Johan,

Hope all the nodes are able to communicate with each other.Please try to ping from one node to other & check the response.If it can talk to each other then stop all the nodes,Clear pr_sys_statusnodes table & restart server node by node.First start util nodes followed by stream node & then web nodes.

Please let me know If issue still persists after that.

 

Thanks,
Abhinav
Posted: 2 years ago
Posted: 2 Dec 2019 1:41 EST
Johan Hoogenboezem (JohanH55) Ernst & Young Senior Manager
Ernst & Young
ZA

Hi Abhinav

Sorry it took so long to get back to this. Hectic weekend with Black Friday and all. We have permission to restart all the nodes tonight. I will let you know if it helped. All the nodes can ping each other.

Regards,

Johan
Posted: 2 years ago
Posted: 4 Dec 2019 13:11 EST
Abhinav Gupta (Abhinav7) PEGA Manager, Technical Support - Platform Service Engineering
Pegasystems Inc.
IN

Hi Johan,

Can you please provide an update.Did it work?

Thanks,

Abhinav
Posted: 2 years ago
Posted: 5 Dec 2019 1:22 EST
Johan Hoogenboezem (JohanH55) Ernst & Young Senior Manager
Ernst & Young
ZA

Hi Abhinav

The restart was postponed to last night to coincide with other downtime. We took the nodes down and when we looked at the table it was empty. We brought the nodes back up and the Kafka server is still down. The four records in the table are back. So no, it didn't work.

Regards,

Johan
Posted: 2 years ago
Posted: 5 Dec 2019 14:25 EST
Abhinav Gupta (Abhinav7) PEGA Manager, Technical Support - Platform Service Engineering
Pegasystems Inc.
IN

Hi Johan,

Okay, It means something wrong with kafka itself.Can you please share error logs which got generated during kafka startup.

Thanks,
Abhinav
Posted: 2 years ago
Posted: 6 Dec 2019 1:37 EST
Johan Hoogenboezem (JohanH55) Ernst & Young Senior Manager
Ernst & Young
ZA

Hi Abhinav

In my very first post I included an extract from that log file. I'm attaching a copy of the latest log file.

Regards,

Johan

Accepted Solution

Posted: 2 years ago
Updated: 2 years ago
Posted: 13 Mar 2020 2:03 EDT
Updated: 6 May 2020 7:55 EDT
Johan Hoogenboezem (JohanH55) Ernst & Young Senior Manager
Ernst & Young
ZA

I was instructed by Pega support in response to SR-D67690 to place the following in prconfig of the problem node:

<env name="dsm/services/stream/server_properties/broker.id" value="8">

As far as I could gather this is supposed to force the broker id. It wasn't sufficient, though. The node kept on starting up with the wrong id. I eventually traced it to some db table that still had an entry for the node with the wrong broker id. After I deleted that the Kafka node started functioning correctly again. It resolved a LOT of stability issues. Unfortunately I do not recall the name of the db table and I no longer have contact with the client. I don't think it was pr_sys_statusnodes, though.
Posted: 2 years ago
Posted: 17 Mar 2020 13:27 EDT
Marissa Rogers (MarissaRogers) MOD Senior Knowledge Management Specialist
Pegasystems Inc.
US

Hi @JohanH55,

I've updated your thread to include your SR details. I also updated your Products and Versions based on that SR.

If the Versions are not correct, please click the Edit icon and update them to the proper versions.

Thank you for using the Pega Collaboration Center!

Marissa | Senior Moderator | Pega Collaboration Center
Posted: 2 years ago
Posted: 21 May 2020 18:50 EDT
Ravibabu Gudipudi (RaviBabuG) Standard and Poors CLSA
Standard and Poors
US

We have received below exceptions while restarting one of the node and  restart is failing continuously due to kafka exceptions .

ERROR Error while creating ephemeral at /brokers/ids/1005, node already exists and owner '-23442432' does not match current session '1235353453576456' (kafka.zk.KafkaZkClient$CheckedEphemeral).

ERROR [KafkaServer id=1004] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists   at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38) [kafka_2.11-1.1.0.4.jar

Solution : 

Deleted the node related information from below tables then restarted the node & its started fine

DELETE FROM pr_sys_statusnodes where PYNODENAME='node ID'

DELETE FROM  pr_data_stream_nodes where PYNODEID like '/brokers/ids/1005%

We'd prefer it if you saw us at our best.

Pega Collaboration Center has detected you are using a browser which may prevent you from experiencing the site as intended. To improve your experience, please update your browser.

Close Deprecation Notice