Dedicated Queue Processor connecting to AWS S3 drops empty and multi req/res in one file
I'm using a Queue processor to send an incoming JSON file via a REST service in Pega to AWS bucket. When I run 1 test at a time I only see the request and response pertaining to that particular request but when I run these requests in bulk say 20/50/100 requests at a time then I see some files are empty and some have multiple request/response put together in one file.
Call hierarchy is like this..
External App ---posts----> REST service running in Pega ----> Activity (process the request and generate the response)
So in the Activity before it sends the response a dedicated Queue Processor is called which calls a Dataset that uses repository that connects to AWS bucket.
The dedicated Queue Processor is configured as a background processor to process immediately with 5 threads per node.
What is expected: The json file dropped in AWS bucket should only have the appropriate request & response json associated with that request.
What is happening: I see empty json files and files with multiple request/response in the same file
Is this happening because Pega runs the queue processor in parallel threads(configured to 5) per node and somehow it puts the record into one file and sends the other one empty?
Is there any configuration that I can do to avoid this empty and multiple req/res in the same file?
I'm using a Queue processor to send an incoming JSON file via a REST service in Pega to AWS bucket. When I run 1 test at a time I only see the request and response pertaining to that particular request but when I run these requests in bulk say 20/50/100 requests at a time then I see some files are empty and some have multiple request/response put together in one file.
Call hierarchy is like this..
External App ---posts----> REST service running in Pega ----> Activity (process the request and generate the response)
So in the Activity before it sends the response a dedicated Queue Processor is called which calls a Dataset that uses repository that connects to AWS bucket.
The dedicated Queue Processor is configured as a background processor to process immediately with 5 threads per node.
What is expected: The json file dropped in AWS bucket should only have the appropriate request & response json associated with that request.
What is happening: I see empty json files and files with multiple request/response in the same file
Is this happening because Pega runs the queue processor in parallel threads(configured to 5) per node and somehow it puts the record into one file and sends the other one empty?
Is there any configuration that I can do to avoid this empty and multiple req/res in the same file?
Setting 1 thread per node in the Queue processor works fine, Pega always produces the json file with 1 request & response but this beats the advantage of performance optimization.
Is there any other way this can be achieved?