Discussion

1
Replies
135
Views
ChakriP8 Member since 2014 62 posts
Accenture
Posted: June 11, 2016
Last activity: June 27, 2016
Closed

Design Question for processing large data

This is more of a design question and your comments would be appreciated.

I am writing an Agent which needs to process about 300K users. For each of the users, my requirement is to enter a flow and create WorkObjects.

I have the users in a custom employee table in a different DB which I query to bring the user data back.

When I query this table, my query does a ' select * '  to select ALL the employees. I put this received data on the clipboard and the subsequent Agent Activities loop through each of the records (pxResults) on the clipboard to enter the flow and create the workobjects.

The question i have is it is possible to select about 3K records or less at once, process these employee records, select the next 3K records (so that the clipboard NOW has only these 3K employee records) process them and so on.

If possible to accomplish this HOW would I go about it. Any code examples would be extremely helpful.

Data Integration
Moderation Team has archived post
Share this page LinkedIn