satsih chitturi (satsihc6)
JP Morgan Chase

JP Morgan Chase
satsihc6 Member since 2017 14 posts
JP Morgan Chase
Posted: July 4, 2020
Last activity: July 5, 2020
Posted: 4 Jul 2020 19:08 EDT
Last activity: 5 Jul 2020 18:56 EDT

Data Flows- Partitioning Key

We have a huge data loaded into DDS table and from that source another DF is reading it. We have configured 20 Threads for the Data flow and have 2 Batch Nodes with created 80 Partitions. But issue is when the DF is processing its processing all the records in single partition which is taking time to process all the records.

Can some one please help on how to distribute the data across all the partitions during the DF processing when the source is DDS. If the source is Database Table mapped dataset then we can configure the partition key but not having that option for the DDS.




Pega Platform 8.2.2 Data Integration Java and Activities Financial Services Lead System Architect