Discussion
Introducing Delay in File Listener
I know there has been a few discussions in the PDN about how a file listener (only one running on one node) would process the files if there are multiple files placed in the folder. My questions is whether it is possible to introduce a delay between the processing of multiple files in a file listener.
Scenario:5 files are put in the folder which is monitored by the file listener. Each file has a specific number of records. I would like to have the processing sequence as follows:
- FL1 picked up and the service file processes the records.
- there is a gap of a minute or so then FL2 is picked up for processing and so on
Is this practically possible in Pega? I have tried using the Idle Processing section but to no avail.
the only option I see is to have a job running on the OS where Pega is installed, and this job would send the file one at a time to the folder which Pega is monitoring.
any light in this direction would be very appreciated.
I don't understand the requirement very well. Do these files need to be processed in a certain order and does file listener need to apply different logic on different files?
Hi Anoojit,
Wai-kei is right there is nothing in the file listener code to configure a delay. The file listener is greedy and will continue processing until all matching files are gone.
Just for completeness can you tell us what your PRPC version is?
I also have a question regarding the use case. It is not every day we get a request to slow things down. Can you talk more about that as well?
Hi,
The requirement is that the folder which Pega listens to would be filled by an external application. That application would pump in files at regular intervals and it may turn out that at a particular point in time there are more than 1 file in the folder. Our application interacts with a plethora of external systems which are load sensitive, so if we pump in large number of requests from multiple files into them , they would potentially become unresponsive. I understand that this limitation / restriction is on the external system, and not Pega, but I was looking for a mechanism by which I can say that I will process each file after an interval od 1 minute between them so that the load is regulated.
We are using Pega 7.1.7
However I did see that if I put in a Wait method in the Idle Processing, there is indeed a small delay (whatever I put as waitseconds) between processing of the file contents. But I am not sure if it is a good practice to have waiting threads in the system, specially if tomorrow we have 5 more listeners with similar requirements.
Here at Pega, we depend on an internal app (are we allowed to say its name here ?) that keeps track of all customer cases. That app depends on many of Pega’s core features, so when I’m wondering how to do something specific, I often try to think examples where that app already does it. (I also often think about the dark side which is, when someone mentions a suspected bug and I’m wondering how to reproduce it, I think about situations where I may have encountered that bug in our internal app).
That internal app generates a note on a case that contains email that is received for the case (identified by the case number appearing in the subject line of the email).
If the case is locked, and the app can not attach the note, it waits several minutes before trying again. Let’s think about how it orchestrates this waiting several minutes and trying again (or merely ask the developers how they do it!), since it seems to me that that waiting is exactly what you’re asking for in this discussion. In your situation, instead of a case being locked being your signal to wait, it sounds like you merely want to look at the system clock, and if not enough time has passed yet, you want to wait.
/Eric
You should be able to throttle the speed at which File Listener processes the files using
|
Polling Interval (seconds) |
Enter the number of seconds this listener waits before it checks for newly arrived files. |
and Idle Processing
|
Activity Class |
Optional. To identify an activity that runs after the processing of a file is complete, enter the Applies To key part here. |
Increase the polling interval and specify an activity that runs after every file is processed.
Hi Vipin,
Thanks for the post. Actually I was kind of able to control the pace at which the file listener picks up and process the file with the idle processing. I introduced a wait on the thread for a definite period of time.
Hi Anoojit,
Looking at the FileListener code now and unfortunately i don't see any logic to put a delay in between processing of each file. Basically we just loop through each matching file in the directory and process the file one at a time without any delay. As you mentioned above, controlling it from the OS level may be the way to go for now.
=waikei