Here are some of the factors, you should consider while tuning the rule cache (list is not precise)
1. Check about different types of work objects
2. Consider the reports in your application which might be used frequently
3. Check any defined agent processing has occurred and that all require service and connect rules have been used.
4. Consider any other rule processing that will occur in normal use.
While defining the rule cache limit, make sure all the rules have been rule assembled(run all rule resolved rules) at least once.This will help PRPC to build the rules assembly cache. Once the system has exercised all the main line rules, accommodate for clearing rule cache using SMA.
Perform the above mentioned activity once again and run the system for good amount of time. Now, launch the rule cache summary page and spot the numbers mentioned in Database Cache Summary(mainly Instances now). This will give the approximate size of the rule cache required for the data base.
Once the number is identified in SMA, as per PEGA recommendation the Instancecountlimit in environmental variables should be 1.5 to 2 times of the actual number(Instance value).
Hope this information helps you in identifying correct cache limit.
The information provided in this post is good but tuning caches is a delicate matter - SME's should be involved to ensure the best possible outcome is obtained. If bugs or other serious issues are encountered then of course GCS should be involved.