Offline Data Pages do not accept Params: need advice.
The customer has a requirement to set up offline data on mobile devices (both iOS and Android).
The backend data set is very large (380k [more? records]; probably too large for performance reasons on the back-end PRPC system and probably too large for the Mobile Device storage.
Previously: Parameters were specified on the Data Page in an attempt to cut down the recordset to a few hundred records.
This approach failed (the sync process would 'hang'): and Pega SMEs were engaged.
SMEs identified the reason: Parameterized Data Pages cannot be used in this context (the params are ignored, or not available in this context).
Therefore the entire data-set was downloaded to the mobile device - unfiltered. (At least on the 'initial sync').
The solution depends on getting the data that is actually needed to be worked on.
For example in this case instead of getting all the flight data for all the passengers for 2 years down to the client for a crew member to access in offline, we need to refine the filter criteria at point of retrieval on server during the packaging sync request::
1. Add a new column for the crew member to the flight master data table (if possible) so the sync gets filtered data only for the flight manifests of which the crew member is part of. It may not be possible to change existing tables / schema. Instead we can do a Join on the crew member roster with the flight passenger data .
2. Add the condition to the report definition to essentially get only data for the current day (or maybe day before and day ahead) so that crew member gets immediately relevant data . 300 * 200 = 60000 rows.
Once we get that manageable subset we may use large data pages to get data for a particular flight when we run the case.
The issue here is how to optimize the data page population itself. The unparameterized master data page has close to 4.4 million records. This has to be brought down to a manageable size before marking it as a large data page.
In such cases it is best to use the operator/user context to retrieve only those relevant records. Then the large data page operations can kick in and make the operation on the data set smoother.
Create a new datapage, and source it from an Activity. In the activity, load data from the original datapage (the huge one) and filter it whatever way you like (e.g. in a Java step). No need to pass any parameters = no problems.
Use that new datapage as a source for your UI controls (lists, forms, etc.). Behind the scenes it will make use of the huge one and filter it.
Sounds like this should work : would this require us to create one datapage for every unique set of params though ?
So if we had (say) two parameters 'name' and 'city' : and one big datapage on the backend which associates names with cities ; and we want to fetch only the rows where 'name=joe' and city='new york'; would we need a datapage for this combination : something like D_MyPage_joe_new_york ?
EDIT: Or does the OOTB 'sync' process already carry a unique identified for the device of some sort ? If there is such a 'device_id' - we could (with some customization) use that as a 'namespace' for unique params.
So I mean: lets say our device has an ID of 'device123': and this particular device is interested in only the subset of data where 'name=joe, city=new york' - but we are unable to pass the parameters directly in the sync request; so *maybe* we could send up another request to PRPC -uploading our parameters 'name=joe, city=new york' to *another* datapage which indexed by our 'deviceID' 'device123' ; *before* the sync is made.
That way our filtering Activity (presuming this also has some visibility of such a 'deviceid') could look up the parameters that way.
But there is a lot of hypotheticals in that guess above.....