To do this, the developer must create an Authentication Service (Org & Security -> Authentication -> Create Authentication Service), and here the developer provides an Authentication Activity. PRPC comes with an example activity called Code-Security . IACAuthentication. This activity can be saved into a custom ruleset and then modified for the authentication needed, as required.
However, for your custom authentication activity to be usable it must be callable it must be accessible from the BROWSER requestor type for an unauthenticated user. To do this, open the BROWSER requestor Type from 'Records->SysAdmin->Requestor Type', and you will see that it uses Access group 'PRPC:Unauthenticated'. Open this access group, and you will see it is built on Application 'PegaRULES:07.10'.
We cannot edit the BROWSER requestor type to use a different access group, we just get the error message "You are not authorized to create, modify, or lock instance DATA-ADMIN-REQUESTOR PRPC!BROWSER'.
We cannot edit the PRPC:Unauthenticated access group to use a different application or to include a new production ruleset, we just get the error "You are not authorized to create, modify, or lock instance DATA-ADMIN-OPERATOR-ACCESSGROUP PRPC!UNAUTHENTICATED".
We cannot edit the application PegaRULES:07.10 to add a new ruleset, we just get the error "Supply password to update: Incorrect password specified" (we have tried all the obvious passwords, install, rules, etc).
So, how to modify a rule for the BROWSER requestor type to be able to run, by an unauthenticated user?
The system name "prpc" is protected - when you use this value you are not able to update certain records (including Data-Admin-Requestor instances and others).
You should update the Dynamic System Setting (Pega-Engine) prconfig/identification/systemName/default - altering the value from "prpc" to some local name (you can even use the "pega" value which has records shipped OOTB).
Once you make this DSS change and restart, you should be able to update the BROWSER D-A-Requestor instance for your altered system name.
I tried this suggestion - disaster! After editing that DSS setting and then rebooting my system, the system is now completely uncontactable. I just get an http:503 (service unavailable) error response in my browser when I try and log on to it. Restoring from backup - thank goodness it is just a PVS VMware image.
This issue was solved for me by downloading and installing a new PVS system, this time one that runs on VirtualBox (the old one used VMWare). I noticed with the new system, the Dynamic System Setting 'prconfig/identification/systemName/default' was already set to "pega", and sure enough when I tried to edit the BROWSER D-A Requestor instance, it worked ootb and I had no trouble making the change I needed.
So I guess Nick's answer above is the correct one. Thanks, Nick.