Posted: 11 Jun 2016 12:09 EDT Last activity: 27 Jun 2016 21:17 EDT
Design Question for processing large data
This is more of a design question and your comments would be appreciated.
I am writing an Agent which needs to process about 300K users. For each of the users, my requirement is to enter a flow and create WorkObjects.
I have the users in a custom employee table in a different DB which I query to bring the user data back.
When I query this table, my query does a ' select * ' to select ALL the employees. I put this received data on the clipboard and the subsequent Agent Activities loop through each of the records (pxResults) on the clipboard to enter the flow and create the workobjects.
The question i have is it is possible to select about 3K records or less at once, process these employee records, select the next 3K records (so that the clipboard NOW has only these 3K employee records) process them and so on.
If possible to accomplish this HOW would I go about it. Any code examples would be extremely helpful.