Now, we also have another Pega Platform environment which is version 7.4. When I looked at the same configuration in Stream Service, a node was not even existent (I never did it because I thought this menu is for Decisioning - we are only using Pega Platform). However, Search seems to be working fine.
"Stream Service has to be running in order for Search to work from version 8.1, but in prior version, it is not required." Is this statement correct? If yes, why is that so?
From Pega 8.1 and later releases, Search runs through the Queue Processor (that is, Kafka and Real-time Decisioning Services). Therefore, you need to enable the Stream service on the Services landing page.
#1. This Stream Service menu is in "Decisioning", and I see it as very confusing because people who are using only Pega Platform would not see this menu carefully. Search is a very basic Pega Platform functionality and I would suggest that to be outside of Decisioning menu. Would you agree?
#2. Could you also give an explanation about what technology Search runs on in prior version (as opposed to Kafka and Real-time Decisioning Services)?
The Stream service was originally used by Decisioning and not by the base platform, so moving it out from that location would confuse people who were already using it regularly, so I think either way some people would have ended up being confused by its location. Changing it now may just cause more confusion.
Prior to using Kafka, search was utilizing agents and the database. Queue Processors utilizing Kafka are able to perform their processing more quickly.
Thanks Nick for your comment, it is very helpful. I understand this way - Search functionality used to utilize agents and database in 7x, and it now utilizes Queue Processor (that is, Kafka and Real-time Decisioning Services) in 8x.
Here is my guess to my own question - In 7.4, Stream Service was introduced but Search was not utilizing it at that point (still utilizing Agent). So in 7.4 even if you do not run Stream Service, Search still works fine. Beginning with 8x, Search started to use Stream Service.
If my hypothesis above is correct, it explains what I am observing. Can someone let me know if this is fair or not?
Appreciate your observation, It is completely right. Let me give you a brief overview.
From pega 8.1 onwards, Job scheduler & QueueProcessor have been introduced. Pega platform 8.1 includes three default queue processor rules.
pyFTSIncrementalIndexer, which replaces agent FTSIncrementalIndexer and performs incremental (search) indexing in the background. This Queue Processor rule posts Rule, Data and Work objects into the Search subsystem, as soon as we create or change them. This helps to keep search data current and reflects the content of the database closely. Therefore you couldn't search in 8.x when the stream node is down while it works in 7.4 without stream node.
If you need more details then please watch the video mentioned in the below link. It covers all the key concepts & demonstrates search with & without stream node in 8.x version.