Posted: 12 Dec 2020 13:28 EST Last activity: 14 Dec 2020 8:38 EST
an external database/system holding 50 million record - how to Show them on pega screen
my requirement is below .. please suggest a best fit solution without effecting the system performance while loading data.
an external database ( with RDB calls or OBJ calls) /external system Services (REST svc calls) ... etc.. holding 50 million records and i have to Show them on pega screen .. what is the best possible solution can any one suggest without degrading effecting the system performance
***Edited by Moderator: Pooja Gadige to add platform capability tag***
this is could be a hypothetical situation (was a design approach question asked by a colleague)
50 million reside in the external system and we want to show it in pega , if we want to show it on UI is pagination feature configurable here .. as per my understanding so far .. though pagination shows data in multiple pages to user , in reality loads all the records in the clipboard .. which may not be a good idea.
and your last question i am not sure if it makes sense , so my design approaches are processing at the back end
OPTION #1 Run a std agent /queue scheduler in the background and create a report using separate reporting database and using separate Node
and use either REST call or mapping external DB into pega classes and
use obj-browse in each call get the total count of records currently available in the table and count of current received records , ordered by row id
and in subsequent calls pass the query with rowid '>' last received rowid, while sum the count of records received so far in and minus from total records available in the external system ( we cannot put any flag of processed records in external system ) until there are no more records to be processed
OPTION #2 Run BIX extraction tool (but do we need to first load data into pega Database , so that we can use BIX?)
OPTION #3 use external BI tools
would be happy to be corrected in any of the approaches and open for new thoughts.