Chakri Payyavula (ChakriP8)

ChakriP8 Member since 2014 65 posts
Posted: June 11, 2016
Last activity: June 27, 2016
Posted: 11 Jun 2016 12:09 EDT
Last activity: 27 Jun 2016 21:17 EDT

Design Question for processing large data

This is more of a design question and your comments would be appreciated.

I am writing an Agent which needs to process about 300K users. For each of the users, my requirement is to enter a flow and create WorkObjects.

I have the users in a custom employee table in a different DB which I query to bring the user data back.

When I query this table, my query does a ' select * '  to select ALL the employees. I put this received data on the clipboard and the subsequent Agent Activities loop through each of the records (pxResults) on the clipboard to enter the flow and create the workobjects.

The question i have is it is possible to select about 3K records or less at once, process these employee records, select the next 3K records (so that the clipboard NOW has only these 3K employee records) process them and so on.

If possible to accomplish this HOW would I go about it. Any code examples would be extremely helpful.

Data Integration
Moderation Team has archived post, This thread is closed to future replies. Content and links will no longer be updated. If you have the same/similar Discussion, please write a new Discussion.